Image, video quality identification method, model training method, device and equipment

Abstract

The disclosure provides an image, a video quality identification method, a model training method, an apparatus, a device, a storage medium and a computer program product, relates to the technical field of data processing, and in particular to the fields of video quality identification and image quality identification. The specific implementation scheme of the video quality identification method is as follows: obtaining a to-be-identified video, extracting at least two to-be-identified images from the to-be-identified video; performing equal proportion scaling processing on each to-be-identified image to obtain a size-transformed image corresponding to each to-be-identified image; extracting image features from the to-be-identified image and the corresponding size-transformed image respectively, fusing the extracted image features to obtain a video feature of the to-be-identified video; and identifying the video quality of the to-be-identified video according to the video feature.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of video quality recognition and image quality recognition. Background Technology

[0002] In existing technologies, video quality recognition typically involves extracting images from a video, unifying each image to a fixed size, inputting them into a convolutional neural network, and then having the convolutional neural network identify the image quality score of each image. The average of all the image quality scores is then used as the video's image quality score. However, the accuracy of this recognition method is limited. Summary of the Invention

[0003] This disclosure provides an image and video quality recognition method, a model training method, an apparatus, a device, a storage medium, and a computer program product.

[0004] According to a first aspect of this disclosure, a method for identifying video quality is provided, comprising:

[0005] Obtain the video to be identified, and extract at least two frames of images to be identified from the video to be identified;

[0006] Each frame of the image to be identified is scaled proportionally to obtain a size-transformed image corresponding to each frame of the image to be identified.

[0007] Image features are extracted from the image to be identified and the corresponding size-transformed image, respectively, and the extracted image features are fused to obtain the video features of the video to be identified;

[0008] The video quality of the video to be identified is determined based on the video features.

[0009] According to a second aspect of this disclosure, a method for training a video quality recognition model is provided, comprising:

[0010] Obtain video samples labeled with video quality annotation information, and extract at least two frame image samples from the video samples;

[0011] Each frame image sample is scaled proportionally to obtain a size-transformed image sample corresponding to each frame image sample.

[0012] The image samples and corresponding size-transformed image samples are input into a video quality recognition model, which outputs video quality prediction information of the video samples based on video features; the video features are obtained by fusing image features extracted from the image samples and corresponding size-transformed image samples.

[0013] The loss error is calculated based on the video quality prediction information and the video quality annotation information, and the model parameters of the video quality recognition model are adjusted based on the loss error.

[0014] According to a third aspect of this disclosure, a method for identifying image quality is provided, comprising:

[0015] Acquire the image to be recognized;

[0016] The image to be recognized is scaled proportionally to obtain a size-transformed image of the image to be recognized;

[0017] Image features are extracted from the image to be identified and the corresponding size-transformed image, respectively, and the extracted image features are fused to obtain the image fusion features of the image to be identified;

[0018] The image quality of the image to be identified is determined based on the image fusion features.

[0019] According to a fourth aspect of this disclosure, a method for training an image quality recognition model is provided, comprising:

[0020] Obtain image samples labeled with image quality annotation information;

[0021] Each image sample is scaled proportionally to obtain a size-transformed image sample corresponding to each image sample.

[0022] The image sample and the corresponding size-transformed image sample are input into the image quality recognition model, so that the image quality recognition model outputs image quality prediction information based on the image fusion features of the image sample; the image fusion features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image.

[0023] The loss error is calculated based on the image quality prediction information and the image quality annotation information, and the model parameters of the image quality recognition model are adjusted based on the loss error.

[0024] According to a fifth aspect of this disclosure, a video quality recognition device is provided, comprising:

[0025] The video acquisition module is used to acquire the video to be identified and extract at least two frames of images to be identified from the video to be identified.

[0026] The scaling module is used to scale each frame of the image to be recognized proportionally to obtain a size-transformed image corresponding to each frame of the image to be recognized.

[0027] The feature extraction module is used to extract image features from the image to be identified and the corresponding size-transformed image, respectively, and fuse the extracted image features to obtain the video features of the video to be identified;

[0028] The quality recognition module is used to identify the video quality of the video to be recognized based on the video features.

[0029] According to a sixth aspect of this disclosure, a training apparatus for a video quality recognition model is provided, comprising:

[0030] The video acquisition module is used to acquire video samples labeled with video quality annotation information and extract at least two frame image samples from the video samples.

[0031] The scaling module is used to scale each frame image sample proportionally to obtain a size-transformed image sample corresponding to each frame image sample.

[0032] The input module is used to input the image sample and the corresponding size-transformed image sample into the video quality recognition model, so that the video quality recognition model outputs video quality prediction information of the video sample based on video features; the video features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image sample.

[0033] The training module is used to calculate the loss error based on the video quality prediction information and the video quality annotation information, and to adjust the model parameters of the video quality recognition model based on the loss error.

[0034] According to a seventh aspect of this disclosure, an image quality recognition device is provided, comprising:

[0035] The image acquisition module is used to acquire the image to be recognized;

[0036] The scaling module is used to perform proportional scaling on the image to be recognized to obtain a size-transformed image of the image to be recognized.

[0037] The feature extraction module is used to extract image features from the image to be identified and the corresponding size-transformed image, respectively, and fuse the extracted image features to obtain the image fusion features of the image to be identified;

[0038] The quality recognition module is used to identify the image quality of the image to be recognized based on the image fusion features.

[0039] According to the eighth aspect of this disclosure, a training apparatus for an image quality recognition model is provided, comprising:

[0040] The image acquisition module is used to acquire image samples labeled with image quality annotation information;

[0041] The scaling module is used to scale each image sample proportionally to obtain a size-transformed image sample corresponding to each image sample.

[0042] The input module is used to input the image sample and the corresponding size-transformed image sample into the image quality recognition model, so that the image quality recognition model can output image quality prediction information based on the image fusion features of the image sample; the image fusion features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image.

[0043] The training module is used to calculate the loss error based on the image quality prediction information and the image quality annotation information, and to adjust the model parameters of the image quality recognition model based on the loss error.

[0044] According to a ninth aspect of this disclosure, an electronic device is provided, comprising:

[0045] At least one processor; and

[0046] A memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0048] According to a tenth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any one of the preceding claims.

[0049] According to the eleventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0050] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0051] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0052] Figure 1 A flowchart illustrating a video quality recognition method provided in this embodiment of the disclosure;

[0053] Figure 2A flowchart illustrating image feature extraction in a video quality recognition method provided in this embodiment of the disclosure;

[0054] Figure 3 A flowchart illustrating the acquisition of image fusion features in a video quality recognition method provided in this embodiment of the disclosure;

[0055] Figure 4 A flowchart illustrating a training method for a video quality recognition model provided in this embodiment of the disclosure;

[0056] Figure 5 A flowchart illustrating an image quality recognition method provided in this embodiment of the disclosure;

[0057] Figure 6 A flowchart illustrating a training method for an image quality recognition model provided in this embodiment of the disclosure;

[0058] Figure 7a A schematic diagram of a video quality recognition device provided in an embodiment of this disclosure;

[0059] Figure 7b A schematic diagram of a training device for a video quality recognition model provided in an embodiment of this disclosure;

[0060] Figure 7c A schematic diagram of a module for an image quality recognition device provided in an embodiment of this disclosure;

[0061] Figure 7d A schematic diagram of a training device for an image quality recognition model provided in an embodiment of this disclosure;

[0062] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0063] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0064] Figure 1 A flowchart illustrating a video quality identification method provided in this disclosure embodiment, the video quality identification method comprising the following steps:

[0065] Step 101: Obtain the video to be identified and extract at least two frames of images to be identified from the video.

[0066] The number of images to be identified extracted from the video to be identified and the frame extraction frequency can be set according to the actual situation; the images to be identified can be extracted at equal intervals or at unequal intervals; the embodiments disclosed herein do not particularly limit this.

[0067] Step 102: Perform proportional scaling on each frame of the image to be recognized to obtain a size-transformed image corresponding to each frame of the image to be recognized.

[0068] In one embodiment, the image S to be recognized is enlarged proportionally according to a first scaling ratio to obtain an enlarged size-transformed image S1.

[0069] The number of first scaling ratios can be set according to the actual situation. When a first scaling ratio is set, a magnified size-transformed image S1 is obtained. Subsequently, the video quality of the video to be identified is identified based on the image to be identified S and a size-transformed image S1. When multiple different first scaling ratios are set, multiple magnified size-transformed images S1 are obtained. Subsequently, the video quality of the video to be identified is identified based on the image to be identified S and multiple size-transformed images S1.

[0070] The first scaling ratio can be set according to the actual situation. For example, if the first scaling ratio is set to 1.5, the size of the size-transformed image S1 is 1.5 times that of the image S to be recognized; if the first scaling ratio is set to 1.2, the size of the size-transformed image S1 is 1.2 times that of the image S to be recognized.

[0071] In this embodiment of the disclosure, the size-transformed image is obtained by proportional magnification processing, rather than by directly resizing the image to be identified. This avoids distortion of the subject in the size-transformed image due to magnification processing, thereby avoiding affecting the accuracy of video quality recognition in subsequent videos to be identified.

[0072] In one embodiment, the image S to be recognized is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image S2.

[0073] The number of second scaling ratios can be set according to the actual situation. When one second scaling ratio is set, a scaled-down image S2 is obtained. Subsequently, the video quality of the video to be identified is identified based on the image to be identified S and the scaled-down image S2. When multiple different second scaling ratios are set, multiple scaled-down images S2 are obtained. Subsequently, the video quality of the video to be identified is identified based on the image to be identified S and the multiple scaled-down images S2.

[0074] The second scaling factor can be set according to the actual situation. For example, if the second scaling factor is set to 0.5, the size of the size-transformed image S2 is 0.5 times that of the image S to be recognized; if the second scaling factor is set to 0.3, the size of the size-transformed image S2 is 0.3 times that of the image S to be recognized.

[0075] In this embodiment of the disclosure, the size-transformed image is obtained by proportionally reducing the size, rather than directly resizing the image to be recognized. This avoids the deformation of objects in the size-transformed image due to the magnification process, thereby avoiding affecting the accuracy of video quality recognition of the subsequent video to be recognized.

[0076] In one embodiment, the image S to be identified is enlarged proportionally according to a first scaling ratio to obtain at least one enlarged size-transformed image S1, and then reduced proportionally according to a second scaling ratio to obtain at least one reduced size-transformed image S2. Subsequently, the video quality of the video to be identified is determined based on the image S to be identified, the size-transformed image S1, and the size-transformed image S2.

[0077] In this embodiment, the video quality of the video to be identified is determined based on three images of different sizes: the image to be identified (S), the resized image (S1), and the resized image (S2). Because the image features of multiple sizes are referenced, the video quality identification result can accurately evaluate the video quality displayed on screens of different sizes, resulting in higher accuracy. This is because scaling the image to be identified proportionally is equivalent to scaling the video to be identified proportionally, and the scaled video can reflect the video quality of the video to be identified to a certain extent. If the scaled video has good video quality, it indicates that the video quality of the video to be identified is good. Therefore, identifying the video quality of the video to be identified by referencing the image features of multiple sizes can accurately evaluate the video quality displayed on screens of different sizes.

[0078] It should be noted that for different images to be identified, the first scaling ratio of each image to be identified may be the same or different; similarly, the second scaling ratio may be the same or different; the embodiments disclosed herein do not impose any particular limitation on this.

[0079] Step 103: Extract image features from the image to be identified and the corresponding size-transformed image, respectively, and fuse the extracted image features to obtain the video features of the video to be identified.

[0080] If in step 102, the image to be identified is enlarged according to the first scaling ratio to obtain a size-transformed image S1, then in step 103, the video features of the video to be identified are obtained based on the first image features of the image to be identified S and the second image features of the size-transformed image S1. This video feature integrates the image features of two types of images, that is, it takes into account the features of multi-size image granularity, so that the video feature can better characterize the video quality of the video to be identified.

[0081] If in step 102, the image to be identified is scaled down according to the second scaling ratio to obtain a size-transformed image S2, then in step 103, the video features of the video to be identified are obtained based on the first image features of the image to be identified S and the second image features of the size-transformed image S2. This video feature integrates the image features of two types of images, that is, it takes into account the features of multiple image sizes, enabling the video feature to better characterize the video quality of the video to be identified.

[0082] If in step 102, the image to be identified is enlarged according to the first scaling ratio and reduced according to the second scaling ratio to obtain two sizes of size-transformed images, then in step 103, the video features of the video to be identified are obtained based on the first image features of the video image S, the second image features of the size-transformed image S1, and the second image features of the size-transformed image S2. This video feature integrates the image features of three types of images, that is, it takes into account the features of multiple image sizes, enabling the video feature to better characterize the video quality of the video to be identified.

[0083] In one embodiment, the image fusion features of the image to be identified are first obtained based on the first image features of the image to be identified and the second image features of the corresponding size-transformed image, and then the video features of the video to be identified are obtained based on the image fusion features of the image to be identified.

[0084] The following describes one method for obtaining image fusion features of the image to be identified.

[0085] Before obtaining the image fusion features of the image to be identified, it is necessary to extract the first image features and the second image features. See [link to relevant documentation]. Figure 2 Extracting the first image features of the image to be identified includes the following steps:

[0086] Step 103-1: Perform image block processing on the image to be recognized to obtain multiple first image blocks.

[0087] There are no overlapping areas among the first image blocks. The number of first image blocks can be set according to actual conditions, and the sizes of the first image blocks contained in the image to be identified can be the same or different. The image segmentation strategies of the images to be identified can be the same or different. This disclosure does not impose any particular limitations on this.

[0088] Step 103-2: Map each first image block to a first image vector, and map the position information of each first image block to a first position vector.

[0089] In one embodiment, a first image patch is linearly mapped using a fully connected layer to obtain a first image vector and a first location vector. The first image vector and the first location vector have the same dimension, facilitating subsequent feature fusion.

[0090] Step 103-3: Concatenate the first image vector and the first position vector of the first image block to obtain the image block vector of the first image block.

[0091] The first image feature of the image to be identified is determined based on the image block vectors of all first image blocks of the image to be identified.

[0092] The image block vector of each first image block contains the first image vector and the first position vector of the first image block. That is, the image block vector can not only represent the image feature information of the image block, but also the position information of the image block. Based on the information of these two dimensions, the detailed features (first image features) that represent the image quality of the image to be identified can be fully extracted, and thus the video features that represent the video quality can be accurately extracted.

[0093] The specific implementation process of extracting the second image features of a size-transformed image and Figure 2 Similarly, the resized image is first segmented into blocks to obtain multiple second image blocks. Each second image block is then mapped to a second image vector, and the position information of each second image block is mapped to a second position vector. The second image vectors and second position vectors of the second image blocks are concatenated to obtain the image block vector of the second image block. The second image features of the resized image are determined based on the image block vectors of all the second image blocks in the resized image. The specific implementation process will not be elaborated here.

[0094] The image fusion feature of the image to be recognized can be obtained by fusing the first image feature of the image to be recognized and the second image feature of the corresponding size-transformed image. The fusion of the first image feature and the second image feature can be achieved, but is not limited to, through a Transformer encoder.

[0095] It should be noted that, in order to facilitate the subsequent acquisition of video features of the video to be identified, the image segmentation strategy of the image to be identified and the corresponding resized image should preferably be the same. For example, if the image to be identified is divided into 9 first image blocks of equal size, then the corresponding resized image is divided into 9 second image blocks of equal size.

[0096] See Figure 3The following example illustrates another way to obtain image fusion features of an image to be identified, using the image to be identified S and its corresponding size-transformed images S1 and S2 as examples.

[0097] Image blocks are processed for the image to be recognized (S), the scaled image (S1), and the scaled image (S2), resulting in 9 image blocks (a1-a9) for the image to be recognized, 9 image blocks (b1-b9) for the scaled image (S1), and 9 image blocks (c1-c9) for the scaled image (S2), for a total of 27 image blocks. These 27 image blocks are linearly mapped to obtain image block vectors corresponding to each image block. Each image block vector is obtained based on an image vector representing the image feature information of the image block and a position vector representing the position information of the image block. The image block vectors of all image blocks are input into a Transformer encoder for vector fusion to obtain the image fusion features of the image to be recognized.

[0098] The location information of the aforementioned image block includes two parameters: the image identifier to which the image block belongs and the location identifier of the image block in the image. The image identifier is encoded to obtain a first location encoding vector (represented by a rectangle filled with diagonal lines in the figure), and the location identifier is encoded to obtain a second location encoding vector (represented by a rectangle filled with dots in the figure). The location information of an image block is represented by the first location encoding vector and the second location encoding vector.

[0099] Understandably, the first position encoding vector is the same for images belonging to the same image, and different for images. The position identifiers of the three images can be uniformly labeled, for example, the position identifiers of the 27 image blocks can be numbered 1 to 27 sequentially; the 27 image blocks can also be labeled according to the image to which they belong, for example, the 9 image blocks of the image to be identified S can be numbered a1 to a9 sequentially, the 9 image blocks of the scaled image S1 can be numbered b1 to b9 sequentially, and the 9 image blocks of the scaled image S2 can be numbered c1 to c9 sequentially.

[0100] Location identifiers can be encoded using, but is not limited to, the sin-cos position encoding formula, as follows:

[0101] ;

[0102] ;

[0103] in, The dimension of the location encoding vector can be set according to the actual situation, for example, 512; i is an integer, and 1≤i≤512; t represents the location identifier of the image block.

[0104] PE(1,1)=sin(1 / 10000 (2 / 512) () represents the value of the first element in the position vector of the first image patch;

[0105] PE(1,2)=cos(1 / 10000 (2 / 512) ) represents the value of the second element in the position vector of the first image patch.

[0106] The above describes obtaining an image patch vector based on the image vector and the position vector. Specifically, the image vector and the position vector can be concatenated to obtain the image patch vector. Vector concatenation can be vector superposition. If the image vector is a 512-dimensional vector and the position vector is a 512-dimensional vector, the resulting image patch vector is a 1024-dimensional vector. Alternatively, vectors can be added, that is, corresponding elements of the vectors are added. If the image vector is a 512-dimensional vector and the position vector is a 512-dimensional vector, the resulting image patch vector is a 512-dimensional vector.

[0107] To facilitate subsequent fusion, the dimensions of the image vector and the position vector obtained through linear mapping are kept the same.

[0108] It should be noted that, in addition to using a Transformer encoder to fuse image block vectors, the embodiments of this disclosure may also use other methods such as convolutional neural networks to obtain fused image block vectors.

[0109] In one embodiment, the video features of the video to be identified are obtained by fusing the first image features and the second image features through two encoders. Specifically, all image features extracted from the image to be identified and the corresponding size transformation image are first input into the first encoder to obtain the image fusion features of the image to be identified. Then, the image fusion features of all frames of the image to be identified extracted from the video to be identified are input into the second encoder. The video features of the video to be identified are obtained by fusing the image fusion features of each frame of the image to be identified according to the second encoder.

[0110] The second encoder can be a shallow encoder, meaning that the number of network architecture layers of the second encoder is less than the number of network architecture layers of the first encoder.

[0111] In this embodiment of the present disclosure, a first encoder fuses the first image features of the image to be identified and the second image features of the corresponding size-transformed image, and outputs an image fusion feature that can characterize the image quality of the image to be identified. A second encoder fuses the image fusion feature of the image to be identified, and outputs a video feature that can characterize the video quality of the video to be identified. Through the two encoders, video features that accurately evaluate the video quality of the video to be identified on different sized displays can be extracted from multi-sized images. Based on the video features, the accuracy of identifying the video quality of the video to be identified is high.

[0112] Step 104: Identify the video quality of the video to be identified based on its video features.

[0113] Video quality can be characterized, but is not limited to, by sharpness and / or aesthetics.

[0114] When video quality is characterized by sharpness, step 104 identifies the sharpness of the video to be identified based on video features.

[0115] When video quality is characterized by aesthetics, step 104 identifies the aesthetics of the video to be identified based on video features.

[0116] When video quality is characterized by sharpness and aesthetics, step 104 simultaneously identifies the sharpness and aesthetics of the video to be identified based on video features.

[0117] In this disclosure, by referencing the image features of images of multiple sizes, the video quality recognition result can accurately evaluate the video quality displayed on displays of different sizes. Compared with the prior art method of determining the video's clarity / aesthetics by the average of the sharpness / aesthetics of the images extracted from the video, the video quality recognition method provided in this disclosure can more accurately and effectively identify the video quality, and it unifies the standard for video quality evaluation for displays of different sizes, avoiding subjectivity in video quality evaluation.

[0118] Figure 4 The flowchart illustrates a training method for a video quality recognition model provided in this embodiment of the disclosure. The training method for the video quality recognition model includes the following steps:

[0119] Step 401: Obtain video samples labeled with video quality annotation information, and extract at least two image samples from the video samples.

[0120] The number of video samples can be set according to the actual situation. The more video samples there are and the richer the samples are, the more accurate the video quality recognition model can be trained.

[0121] Video quality annotation information is used to evaluate the video quality of video samples. Video quality annotation information may include, but is not limited to, clarity and aesthetics.

[0122] Step 402: Perform proportional scaling on each frame image sample to obtain a size-transformed image sample corresponding to each frame image sample.

[0123] The specific implementation of step 402 is similar to that of step 102, and will not be repeated here.

[0124] Step 403: Input the image samples and the corresponding size-transformed image samples into the video quality recognition model, so that the video quality recognition model can output video quality prediction information of the video samples based on the video features.

[0125] Video features are obtained by fusing image features extracted from image samples and corresponding size-transformed image samples.

[0126] In one embodiment, inputting image samples and corresponding size-transformed image samples into a video quality recognition model includes: performing image block processing on the image samples to obtain at least two first image blocks, performing image block processing on the size-transformed image samples to obtain at least two second image blocks; and inputting both the at least two first image blocks and the at least two second image blocks into the video quality recognition model.

[0127] After image segmentation, the images are input into the video quality recognition model for training. This enables the video quality recognition model to extract more image granular information and learn the ability to accurately and effectively extract video features that characterize the video quality of the video to be recognized.

[0128] In one embodiment, the video quality recognition model includes a fully connected layer, a first encoder, a second encoder, and an output layer; the fully connected layer is used to map each first image block and each second image block into image block vectors respectively; the first encoder is used to extract image features from image samples and resized images, and fuse the extracted image features to obtain image fusion features of the image samples; the second encoder is used to fuse the image fusion features of all image samples extracted from video samples to obtain video features of the video samples; the output layer is used to output image quality prediction information based on the video features.

[0129] The second encoder can be a shallow encoder, meaning that the number of network architecture layers of the second encoder is less than the number of network architecture layers of the first encoder.

[0130] When the video quality annotation information includes sharpness, the image quality prediction information output by the output layer includes sharpness; when the video quality annotation information includes aesthetics, the image quality prediction information output by the output layer includes aesthetics; when the video quality annotation information includes both sharpness and aesthetics, the image quality prediction information output by the output layer includes both sharpness and aesthetics.

[0131] In this embodiment of the disclosure, the video quality recognition model includes two encoders. The two encoders can extract video features from multi-size images to accurately evaluate the video quality of the video to be recognized on different sized displays. This enables the video quality recognition model to accurately and effectively recognize the video quality of the video to be recognized based on these video features.

[0132] Step 404: Calculate the loss error based on the video quality prediction information and the video quality annotation information, and adjust the model parameters of the video quality recognition model based on the loss error.

[0133] The video quality recognition model is iteratively trained until an iteration stopping condition is reached. The trained model can then be used to identify the video quality. The iteration stopping condition may include, but is not limited to, conditions such as the loss error being less than an error threshold or the number of iterations reaching a certain threshold. The error threshold and the number of iterations threshold can be set according to the specific requirements.

[0134] The initial training video quality recognition model can be a model that has been trained using existing technologies. By optimizing the initial training video quality recognition model using the training method provided in this disclosure, a video quality recognition model that can more accurately and effectively identify video quality and has unified the standard for video quality evaluation for displays of different sizes can be obtained. Furthermore, using a model that has been trained can enable the model training to converge quickly and improve the model training speed.

[0135] Figure 5 A flowchart illustrating an image quality recognition method provided in this disclosure embodiment, the image quality recognition method comprising the following steps:

[0136] Step 501: Obtain the image to be recognized.

[0137] Step 502: Perform proportional scaling on the image to be recognized to obtain a size-transformed image of the image to be recognized.

[0138] The specific implementation of step 502 is similar to that of step 102, and will not be repeated here.

[0139] Step 503: Extract image features from the image to be identified and the corresponding size-transformed image respectively, and fuse the extracted image features to obtain the image fusion features of the image to be identified.

[0140] If, in step 502, the image to be identified is enlarged according to the first scaling ratio to obtain a size-transformed image S1, then in step 503, the image fusion feature of the image to be identified is obtained based on the first image feature of the image to be identified S1 and the second image feature of the size-transformed image S1. This image fusion feature integrates the image features of two types of images, that is, it takes into account the features of multi-size image granularity, so that the image fusion feature can better characterize the image quality of the image to be identified.

[0141] If in step 502, the image to be recognized is reduced in size according to the second scaling ratio to obtain a size-transformed image S2, then in step 503, the image fusion feature of the image to be recognized is obtained based on the first image feature of the image to be recognized S and the second image feature of the size-transformed image S2. This image fusion feature integrates the image features of two types of images, that is, it takes into account the features of multi-size image granularity, so that the image feature can better characterize the image quality of the image to be recognized.

[0142] If in step 502, the image to be identified is enlarged according to the first scaling ratio and reduced according to the second scaling ratio to obtain two sizes of size-transformed images, then in step 503, the image fusion feature of the image to be identified is obtained based on the first image feature of the image to be identified S, the second image feature of the size-transformed image S1, and the second image feature of the size-transformed image S2. This image fusion feature integrates the image features of three types of images, that is, it takes into account the features of multi-size image granularity, so that the image fusion feature can better characterize the image quality of the image to be identified.

[0143] The specific implementation method for obtaining image fusion features is similar to that in step 103, and will not be repeated here.

[0144] Step 504: Identify the image quality of the image to be identified based on the image fusion features.

[0145] Image quality can be characterized, but is not limited to, by sharpness and / or aesthetics.

[0146] When image quality is characterized by sharpness, step 504 identifies the sharpness of the image to be identified based on image features.

[0147] When image quality is characterized by aesthetics, step 504 identifies the aesthetics of the image to be identified based on image features.

[0148] When image quality is characterized by sharpness and aesthetics, step 504 simultaneously identifies the sharpness and aesthetics of the image to be identified based on image features.

[0149] In this embodiment, by referencing the image features of images of multiple sizes, the image quality recognition results can accurately evaluate the image quality displayed on screens of different sizes, and a unified standard for image quality evaluation is established for screens of different sizes, avoiding subjectivity in image quality evaluation.

[0150] Figure 6 A flowchart illustrating a training method for an image quality recognition model provided in this disclosure embodiment, the training method comprising the following steps:

[0151] Step 601: Obtain image samples labeled with image quality annotation information.

[0152] The number of image samples can be set according to the actual situation. The more image samples there are and the richer the samples are, the more accurate the image quality recognition model can be trained.

[0153] Image quality annotation information is used to evaluate the image quality of image samples. Image quality annotation information may include, but is not limited to, sharpness and aesthetics.

[0154] Step 602: Scale each image sample proportionally to obtain a size-transformed image sample corresponding to each image sample.

[0155] The specific implementation of step 602 is similar to that of step 102, and will not be repeated here.

[0156] Step 603: Input the image sample and the corresponding size-transformed image sample into the image quality recognition model, so that the image quality recognition model can output the image quality prediction information of the image sample based on the image fusion features of the image sample.

[0157] Image fusion features are obtained by fusing image features extracted from image samples and corresponding size-transformed images.

[0158] In one embodiment, inputting an image sample and a corresponding size-transformed image sample into an image quality recognition model includes: performing image block processing on the image sample to obtain a plurality of first image blocks, performing image block processing on the size-transformed image sample to obtain a plurality of second image blocks; and inputting both the plurality of first image blocks and the plurality of second image blocks into the image quality recognition model.

[0159] After image segmentation, the image is input into the image quality recognition model for training. This enables the image quality recognition model to extract more image granular information and learn the ability to accurately and effectively extract image features that characterize image quality.

[0160] In one embodiment, the image quality recognition model includes a fully connected layer, an encoder, and an output layer; the fully connected layer is used to map each first image block and each second image block into an image block vector respectively; the encoder is used to extract image block features from the image block vectors and fuse the image block features to obtain image fusion features; the output layer is used to output image quality prediction information of image samples based on the image fusion features.

[0161] When the image quality annotation information includes sharpness, the image quality prediction information output by the output layer includes sharpness; when the image quality annotation information includes aesthetics, the image quality prediction information output by the output layer includes aesthetics; when the image quality annotation information includes both sharpness and aesthetics, the image quality prediction information output by the output layer includes both sharpness and aesthetics.

[0162] In this embodiment of the disclosure, the image quality recognition model can extract image features from multi-size images to accurately evaluate the image quality of the image to be recognized on displays of different sizes, so that the image quality recognition model can accurately and effectively recognize the image quality of the image to be recognized based on the image features.

[0163] Step 604: Calculate the loss error based on the image quality prediction information and the image quality annotation information, and adjust the model parameters of the image quality recognition model based on the loss error.

[0164] The image quality recognition model is iteratively trained until an iteration stopping condition is reached. The trained model can then be used to identify the image quality of an image. The iteration stopping condition may include, but is not limited to, conditions such as the loss error being less than an error threshold or the number of iterations reaching a certain threshold. The error threshold and the number of iterations threshold can be set according to the specific circumstances.

[0165] The image quality recognition model for initial training can be a model that has been trained using existing technologies. By optimizing the image quality recognition model provided in this disclosure, an image quality recognition model that can more accurately and effectively identify image quality and has a unified standard for image quality evaluation for displays of different sizes can be obtained. Furthermore, using a model that has been trained can enable the model training to converge quickly and improve the model training speed.

[0166] Corresponding to the aforementioned embodiments of the video / image quality recognition method and the video / image quality recognition model training method, this disclosure also provides embodiments of the video / image quality recognition device and the video / image quality recognition model training device.

[0167] Figure 7a This is a schematic diagram of a video quality recognition device provided as an exemplary embodiment of the present disclosure. The video quality recognition device includes:

[0168] The video acquisition module 71a is used to acquire the video to be identified and extract at least two frames of images to be identified from the video to be identified.

[0169] The scaling module 72a is used to perform proportional scaling on each frame of the image to be recognized to obtain a size transformation image corresponding to each frame of the image to be recognized.

[0170] The feature extraction module 73a is used to extract image features from the image to be identified and the corresponding size transformation image respectively, and fuse the extracted image features to obtain the video features of the video to be identified.

[0171] The quality recognition module 74a is used to identify the video quality of the video to be recognized based on the video features.

[0172] Optionally, when extracting image features from the image to be identified, the feature extraction module is specifically used for:

[0173] The image to be identified is divided into image blocks to obtain at least two first image blocks;

[0174] Each first image block is mapped to a first image vector, and the position information of each first image block is mapped to a first position vector.

[0175] The first image vector and the first position vector of the first image block are concatenated to obtain the image block vector of the first image block; the image features of the image to be identified are determined based on the image block vector of the first image block.

[0176] And / or, when extracting image features from the size-transformed image, the feature extraction module is specifically used for:

[0177] The size-transformed image is divided into image blocks to obtain at least two second image blocks;

[0178] Each second image block is mapped to a second image vector, and the position information of each second image block is mapped to a second position vector.

[0179] The second image vector and the second position vector of the second image block are concatenated to obtain the image block vector of the second image block; the image features of the size-transformed image are determined based on the image block vector of the second image block.

[0180] Optionally, when fusing the extracted image features, the feature extraction module is specifically used for:

[0181] All image features extracted from the image to be identified and the corresponding size-transformed image are input into the first encoder to obtain the image fusion features of the image to be identified;

[0182] The image fusion features of all images to be identified are input into the second encoder to obtain the video features of the video to be identified.

[0183] Optionally, the quality identification module is specifically used for:

[0184] The clarity of the video to be identified is determined based on the video features.

[0185] And / or, identify the aesthetics of the video to be identified based on the video features.

[0186] Optionally, the scaling module is specifically configured to include:

[0187] The image to be recognized is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image.

[0188] And / or, the image to be identified is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

[0189] Figure 7b This is a schematic diagram of a training apparatus for a video quality recognition model provided as an exemplary embodiment of the present disclosure. The training apparatus for the video quality recognition model includes:

[0190] The video acquisition module 71b is used to acquire video samples labeled with video quality annotation information and extract at least two frame image samples from the video samples.

[0191] The scaling module 72b is used to perform proportional scaling on each frame image sample to obtain a size-transformed image sample corresponding to each frame image sample.

[0192] Input module 73b is used to input the image sample and the corresponding size-transformed image sample into the video quality recognition model, so that the video quality recognition model outputs video quality prediction information of the video sample based on video features; the video features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image sample.

[0193] Training module 74b is used to calculate the loss error based on the video quality prediction information and the video quality annotation information, and to adjust the model parameters of the video quality recognition model based on the loss error.

[0194] Optionally, the input module is specifically configured to include:

[0195] The image sample is processed by image block division to obtain at least two first image blocks, and the size-transformed image sample is processed by image block division to obtain at least two second image blocks;

[0196] The at least two first image blocks and the at least two second image blocks are both input into the video quality recognition model.

[0197] Optionally, the video quality recognition model includes a fully connected layer, a first encoder, a second encoder, and an output layer;

[0198] The fully connected layer is used to map each first image block and each second image block into an image block vector, respectively.

[0199] The first encoder is used to extract image features from the image sample and the size-transformed image, and fuse the extracted image features to obtain the image fusion features of the image sample;

[0200] The second encoder is used to fuse the image fusion features of all image samples extracted from the video sample to obtain the video features of the video sample;

[0201] The output layer is used to output video quality prediction information based on the video features.

[0202] Optionally, the scaling module is specifically used for:

[0203] The image sample is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image.

[0204] And / or, the image sample is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

[0205] Figure 7c This is a schematic diagram of a module for an image quality recognition device provided as an exemplary embodiment of the present disclosure. The image quality recognition device includes:

[0206] Image acquisition module 71c is used to acquire the image to be recognized;

[0207] The scaling module 72c is used to perform proportional scaling on the image to be recognized to obtain a size-transformed image of the image to be recognized.

[0208] The feature extraction module 73c is used to extract image features from the image to be identified and the corresponding size-transformed image, respectively, and fuse the extracted image features to obtain the image fusion features of the image to be identified;

[0209] The quality recognition model 74c is used to identify the image quality of the image to be recognized based on the image fusion features.

[0210] Optionally, when extracting image features from the image to be identified, the feature extraction module is specifically used for:

[0211] The image to be identified is divided into multiple first image blocks.

[0212] Each first image block is mapped to a first image vector, and the position information of each first image block is mapped to a first position vector.

[0213] The first image vector and the first position vector of the first image block are concatenated to obtain the image block vector of the first image block; the image features of the image to be identified are determined based on the image block vector of the first image block.

[0214] And / or, when extracting image features from the size-transformed image, the feature extraction module is specifically used for:

[0215] The size-transformed image is divided into multiple second image blocks.

[0216] Each second image block is mapped to a second image vector, and the position information of each second image block is mapped to a second position vector.

[0217] The second image vector and the second position vector of the second image block are concatenated to obtain the image block vector of the second image block; the second image feature of the size-transformed image is determined based on the image block vector of the second image block.

[0218] Optionally, when the image features extracted by fusing are used to obtain the image fusion features of the image to be identified, the feature extraction module is specifically used for:

[0219] The extracted image features are input into the first encoder to obtain the image fusion features of the image to be identified.

[0220] Optionally, the quality identification model is specifically used for:

[0221] The sharpness of the image to be identified is determined based on the image features.

[0222] And / or, identify the aesthetics of the image to be identified based on the image features.

[0223] Optionally, the scaling module is specifically used for:

[0224] The image to be recognized is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image.

[0225] And / or, the image to be identified is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

[0226] Figure 7d This is a schematic diagram of a module for training an image quality recognition model, provided as an exemplary embodiment of the present disclosure. The training device for the image quality recognition model includes:

[0227] The image acquisition module 71d is used to acquire image samples labeled with image quality annotation information;

[0228] The scaling module 72d is used to scale each image sample proportionally to obtain a size-transformed image sample corresponding to each image sample.

[0229] Input module 73d is used to input the image sample and the corresponding size-transformed image sample into the image quality recognition model, so that the image quality recognition model can output image quality prediction information based on the image fusion features of the image sample; the image fusion features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image.

[0230] The training module 74d is used to calculate the loss error based on the image quality prediction information and the image quality annotation information, and to adjust the model parameters of the image quality recognition model based on the loss error.

[0231] Optionally, the input module is specifically used for:

[0232] The image sample is processed by image block division to obtain multiple first image blocks, and the size-transformed image sample is processed by image block division to obtain multiple second image blocks;

[0233] The plurality of first image blocks and the plurality of second image blocks are all input into the image quality recognition model.

[0234] Optionally, the image quality recognition model includes a fully connected layer, an encoder, and an output layer;

[0235] The fully connected layer is used to map the first image block and the second image block into image block vectors, respectively.

[0236] The encoder is used to extract image patch features from the image patch vector and fuse the image patch features to obtain the image fusion features;

[0237] The output layer is used to output image quality prediction information of the image samples based on the image fusion features.

[0238] Optionally, the scaling module is specifically used for:

[0239] The image sample is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image.

[0240] And / or, the image sample is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

[0241] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0242] The technical solutions disclosed herein involve the collection, storage, use, processing, transmission, provision, and disclosure of videos and images, all of which comply with relevant laws and regulations and do not violate public order and good morals.

[0243] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0244] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0245] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0246] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0247] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as image quality recognition methods, video quality recognition methods, image quality recognition model training methods, and video quality recognition model training methods. For example, in some embodiments, any of the above methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of any of the above methods can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform any of the above methods by any other suitable means (e.g., by means of firmware).

[0248] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0249] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0250] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0251] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0252] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0253] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0254] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0255] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for identifying video quality, comprising: Obtain the video to be identified, and extract at least two frames of images to be identified from the video to be identified; Each frame of the image to be identified is scaled proportionally to obtain a size-transformed image corresponding to each frame of the image to be identified. Image features are extracted from the image to be identified and the corresponding size-transformed image, respectively. The image features extracted from all the images to be identified and the corresponding size-transformed images are then fused to obtain the video features of the video to be identified. The video quality of the video to be identified is determined based on the video features.

2. The video quality identification method according to claim 1, wherein, Extracting image features from the image to be identified includes: The image to be identified is divided into image blocks to obtain at least two first image blocks; Each first image block is mapped to a first image vector, and the position information of each first image block is mapped to a first position vector. The first image vector and the first position vector of the first image block are concatenated to obtain the image block vector of the first image block; the image features of the image to be identified are determined based on the image block vector of the first image block. And / or, extracting image features from the size-transformed image, including: The size-transformed image is divided into image blocks to obtain at least two second image blocks; Each second image block is mapped to a second image vector, and the position information of each second image block is mapped to a second position vector. The second image vector and the second position vector of the second image block are concatenated to obtain the image block vector of the second image block; the image features of the size-transformed image are determined based on the image block vector of the second image block.

3. The video quality recognition method according to claim 1, wherein, The fusion of the extracted image features yields the video features of the video to be identified, including: All image features extracted from the image to be identified and the corresponding size-transformed image are input into the first encoder to obtain the image fusion features of the image to be identified; The image fusion features of all images to be identified are input into the second encoder to obtain the video features of the video to be identified.

4. The video quality identification method according to any one of claims 1-3, wherein, The step of identifying the video quality of the video to be identified based on the video features includes: The clarity of the video to be identified is determined based on the video features. And / or, identify the aesthetics of the video to be identified based on the video features.

5. The video quality recognition method according to claim 4, wherein the step of scaling each frame of the image to be recognized proportionally includes: The image to be recognized is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image to be identified is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

6. A training method for a video quality recognition model, comprising: Obtain video samples labeled with video quality annotation information, and extract at least two frame image samples from the video samples; Each frame image sample is scaled proportionally to obtain a size-transformed image sample corresponding to each frame image sample. The image samples and corresponding size-transformed image samples are input into a video quality recognition model, which then outputs video quality prediction information for the video samples based on video features. The video features are obtained by fusing image features extracted from all image samples and corresponding size-transformed image samples of the video samples. The loss error is calculated based on the video quality prediction information and the video quality annotation information, and the model parameters of the video quality recognition model are adjusted based on the loss error.

7. The training method for the video quality recognition model according to claim 6, wherein, The step of inputting the image samples and corresponding size-transformed image samples into the video quality recognition model includes: The image sample is processed by image block division to obtain at least two first image blocks, and the size-transformed image sample is processed by image block division to obtain at least two second image blocks; The at least two first image blocks and the at least two second image blocks are both input into the video quality recognition model.

8. The training method for the video quality recognition model according to claim 7, wherein, The video quality recognition model includes a fully connected layer, a first encoder, a second encoder, and an output layer; The fully connected layer is used to map each first image block and each second image block into an image block vector, respectively. The first encoder is used to extract image features from the image sample and the size-transformed image, and fuse the extracted image features to obtain the image fusion features of the image sample; The second encoder is used to fuse the image fusion features of all image samples extracted from the video sample to obtain the video features of the video sample; The output layer is used to output the video quality prediction information based on the video features.

9. The training method for the video quality recognition model according to any one of claims 6 to 8, wherein the step of scaling each frame image sample proportionally includes: The image sample is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image sample is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

10. A method for recognizing image quality, comprising: Acquire the image to be recognized; The image to be recognized is scaled proportionally to obtain a size-transformed image of the image to be recognized; Image features are extracted from the image to be identified and the corresponding size-transformed image, respectively, and the extracted image features are fused to obtain the image fusion features of the image to be identified; The image quality of the image to be identified is determined based on the image fusion features.

11. The image quality recognition method according to claim 10, wherein, Extracting image features from the image to be identified includes: The image to be identified is divided into multiple first image blocks. Each first image block is mapped to a first image vector, and the position information of each first image block is mapped to a first position vector. The first image vector and the first position vector of the first image block are concatenated to obtain the image block vector of the first image block; the image features of the image to be identified are determined based on the image block vector of the first image block. And / or, extracting image features from the size-transformed image includes: The size-transformed image is divided into multiple second image blocks. Each second image block is mapped to a second image vector, and the position information of each second image block is mapped to a second position vector. The second image vector and the second position vector of the second image block are concatenated to obtain the image block vector of the second image block; the second image feature of the size-transformed image is determined based on the image block vector of the second image block.

12. The image quality recognition method according to claim 10, wherein, The image features extracted by the fusion are used to obtain the image fusion features of the image to be identified, including: The extracted image features are input into the first encoder to obtain the image fusion features of the image to be identified.

13. The image quality recognition method according to any one of claims 10 to 12, wherein, The step of identifying the image quality of the image to be identified based on the image features includes: The sharpness of the image to be identified is determined based on the image features. And / or, identify the aesthetics of the image to be identified based on the image features.

14. The image quality recognition method according to claim 13, wherein the proportional scaling processing of the image to be recognized includes: The image to be recognized is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image to be identified is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

15. A training method for an image quality recognition model, comprising: Obtain image samples labeled with image quality annotation information; Each image sample is scaled proportionally to obtain a size-transformed image sample corresponding to each image sample. The image sample and the corresponding size-transformed image sample are input into the image quality recognition model, so that the image quality recognition model outputs image quality prediction information based on the image fusion features of the image sample; the image fusion features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image. The loss error is calculated based on the image quality prediction information and the image quality annotation information, and the model parameters of the image quality recognition model are adjusted based on the loss error.

16. The training method for the image quality recognition model according to claim 15, wherein, The step of inputting the image samples and corresponding size-transformed image samples into the image quality recognition model includes: The image sample is processed by image block division to obtain multiple first image blocks, and the size-transformed image sample is processed by image block division to obtain multiple second image blocks; The plurality of first image blocks and the plurality of second image blocks are all input into the image quality recognition model.

17. The training method for the image quality recognition model according to claim 16, wherein, The image quality recognition model includes a fully connected layer, an encoder, and an output layer; The fully connected layer is used to map the first image block and the second image block into image block vectors, respectively. The encoder is used to extract image patch features from the image patch vector and fuse the image patch features to obtain the image fusion features; The output layer is used to output image quality prediction information of the image samples based on the image fusion features.

18. The training method for the image quality recognition model according to any one of claims 15 to 17, wherein the step of scaling each image sample proportionally includes: The image sample is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image sample is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

19. A video quality recognition device, comprising: The video acquisition module is used to acquire the video to be identified and extract at least two frames of images to be identified from the video to be identified. The scaling module is used to scale each frame of the image to be recognized proportionally to obtain a size-transformed image corresponding to each frame of the image to be recognized. The feature extraction module is used to extract image features from the image to be identified and the corresponding size-transformed image respectively, and fuse the image features extracted from all the images to be identified and the corresponding size-transformed images to obtain the video features of the video to be identified; The quality recognition module is used to identify the video quality of the video to be recognized based on the video features.

20. The video quality recognition device according to claim 19, wherein, When extracting image features from the image to be identified, the feature extraction module is specifically used for: The image to be identified is divided into image blocks to obtain at least two first image blocks; Each first image block is mapped to a first image vector, and the position information of each first image block is mapped to a first position vector. The first image vector and the first position vector of the first image block are concatenated to obtain the image block vector of the first image block; the image features of the image to be identified are determined based on the image block vector of the first image block. And / or, when extracting image features from the size-transformed image, the feature extraction module is specifically used for: The size-transformed image is divided into image blocks to obtain at least two second image blocks; Each second image block is mapped to a second image vector, and the position information of each second image block is mapped to a second position vector. The second image vector and the second position vector of the second image block are concatenated to obtain the image block vector of the second image block; the image features of the size-transformed image are determined based on the image block vector of the second image block.

21. The video quality recognition device according to claim 19, wherein, When fusing the extracted image features, the feature extraction module is specifically used for: All image features extracted from the image to be identified and the corresponding size-transformed image are input into the first encoder to obtain the image fusion features of the image to be identified; The image fusion features of all images to be identified are input into the second encoder to obtain the video features of the video to be identified.

22. The video quality recognition device according to any one of claims 19-21, wherein, The quality identification module is specifically used for: The clarity of the video to be identified is determined based on the video features. And / or, identify the aesthetics of the video to be identified based on the video features.

23. The video quality recognition device according to claim 22, wherein the scaling module is specifically configured to include: The image to be recognized is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image to be identified is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

24. A training device for a video quality recognition model, comprising: The video acquisition module is used to acquire video samples labeled with video quality annotation information and extract at least two frame image samples from the video samples. The scaling module is used to scale each frame image sample proportionally to obtain a size-transformed image sample corresponding to each frame image sample. The input module is used to input the image sample and the corresponding size-transformed image sample into the video quality recognition model, so that the video quality recognition model outputs video quality prediction information of the video sample based on video features; the video features are obtained by fusing image features extracted from all image samples and the corresponding size-transformed image samples of the video sample. The training module is used to calculate the loss error based on the video quality prediction information and the video quality annotation information, and to adjust the model parameters of the video quality recognition model based on the loss error.

25. The training apparatus for the video quality recognition model according to claim 24, wherein, The input module is specifically used to include: The image sample is processed by image block division to obtain at least two first image blocks, and the size-transformed image sample is processed by image block division to obtain at least two second image blocks; The at least two first image blocks and the at least two second image blocks are both input into the video quality recognition model.

26. The training apparatus for the video quality recognition model according to claim 25, wherein, The video quality recognition model includes a fully connected layer, a first encoder, a second encoder, and an output layer; The fully connected layer is used to map each first image block and each second image block into an image block vector, respectively. The first encoder is used to extract image features from the image sample and the size-transformed image, and fuse the extracted image features to obtain the image fusion features of the image sample; The second encoder is used to fuse the image fusion features of all image samples extracted from the video sample to obtain the video features of the video sample; The output layer is used to output video quality prediction information based on the video features.

27. The training apparatus for the video quality recognition model according to any one of claims 24 to 26, wherein the scaling module is specifically used for: The image sample is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image sample is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

28. An image quality recognition device, comprising: The image acquisition module is used to acquire the image to be recognized; The scaling module is used to perform proportional scaling on the image to be recognized to obtain a size-transformed image of the image to be recognized. The feature extraction module is used to extract image features from the image to be identified and the corresponding size-transformed image, respectively, and fuse the extracted image features to obtain the image fusion features of the image to be identified; A quality identification model is used to identify the image quality of the image to be identified based on the image fusion features.

29. The image quality recognition device according to claim 28, wherein, When extracting image features from the image to be identified, the feature extraction module is specifically used for: The image to be identified is divided into multiple first image blocks. Each first image block is mapped to a first image vector, and the position information of each first image block is mapped to a first position vector. The first image vector and the first position vector of the first image block are concatenated to obtain the image block vector of the first image block; the image features of the image to be identified are determined based on the image block vector of the first image block. And / or, when extracting image features from the size-transformed image, the feature extraction module is specifically used for: The size-transformed image is divided into multiple second image blocks. Each second image block is mapped to a second image vector, and the position information of each second image block is mapped to a second position vector. The second image vector and the second position vector of the second image block are concatenated to obtain the image block vector of the second image block; the second image feature of the size-transformed image is determined based on the image block vector of the second image block.

30. The image quality recognition device according to claim 28, wherein, When the image features extracted by the fusion are used to obtain the image fusion features of the image to be identified, the feature extraction module is specifically used for: The extracted image features are input into the first encoder to obtain the image fusion features of the image to be identified.

31. The image quality recognition device according to any one of claims 28 to 30, wherein, The quality identification model is specifically used for: The sharpness of the image to be identified is determined based on the image features. And / or, identify the aesthetics of the image to be identified based on the image features.

32. The image quality recognition device according to claim 31, wherein the scaling module is specifically used for: The image to be recognized is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image to be identified is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

33. A training device for an image quality recognition model, comprising: The image acquisition module is used to acquire image samples labeled with image quality annotation information; The scaling module is used to scale each image sample proportionally to obtain a size-transformed image sample corresponding to each image sample. The input module is used to input the image sample and the corresponding size-transformed image sample into the image quality recognition model, so that the image quality recognition model can output image quality prediction information based on the image fusion features of the image sample; the image fusion features are obtained by fusing image features extracted from the image sample and the corresponding size-transformed image. The training module is used to calculate the loss error based on the image quality prediction information and the image quality annotation information, and to adjust the model parameters of the image quality recognition model based on the loss error.

34. The training apparatus for the image quality recognition model according to claim 33, wherein, The input module is specifically used for: The image sample is processed by image block division to obtain multiple first image blocks, and the size-transformed image sample is processed by image block division to obtain multiple second image blocks; The plurality of first image blocks and the plurality of second image blocks are all input into the image quality recognition model.

35. The training apparatus for the image quality recognition model according to claim 34, wherein, The image quality recognition model includes a fully connected layer, an encoder, and an output layer; The fully connected layer is used to map the first image block and the second image block into image block vectors, respectively. The encoder is used to extract image patch features from the image patch vector and fuse the image patch features to obtain the image fusion features; The output layer is used to output image quality prediction information of the image samples based on the image fusion features.

36. The training apparatus for the image quality recognition model according to any one of claims 33 to 35, wherein the scaling module is specifically used for: The image sample is enlarged proportionally according to the first scaling ratio to obtain an enlarged size-transformed image. And / or, the image sample is scaled down proportionally according to a second scaling ratio to obtain a scaled-down image.

37. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-18.

38. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-18.

39. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-18.

Images