A data quality assessment method, device, computer and readable storage medium

Abstract

The embodiment of the present application discloses a data quality assessment method, device, computer and readable storage medium, the method comprising: obtaining a media feature map of multimedia data, obtaining at least two data channels for constituting multimedia data; based on a channel attention mechanism, generating a channel attention mask corresponding to at least two data channels, generating a channel feature map of multimedia data according to the channel attention mask and the media feature map; based on a spatial attention mechanism, generating a spatial attention mask corresponding to the channel feature map, generating a spatial feature map of multimedia data according to the spatial attention mask and the channel feature map; outputting the predicted media quality of multimedia data according to the spatial feature map. By adopting the present application, the accuracy of the quality prediction of multimedia data can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data quality assessment method, device, computer and readable storage medium. Background Art

[0002] In the process of video acquisition, processing, transmission and recording, due to the imperfections of imaging systems, processing methods, transmission media and recording equipment, as well as reasons such as object movement or noise pollution, the video will inevitably be distorted and degraded. When recommending videos, the recommendation system needs to filter out videos of poor quality to avoid giving viewers an unpleasant experience. Among them, the video quality assessment algorithm based on reference-free video quality assessment has no reference images, making it difficult to quantify video distortion, degradation and other issues, which brings great challenges to quality assessment. The video quality assessment algorithm without reference refers to an algorithm that directly evaluates the quality of any given video segment without the help of the original lossless reference video information. Among them, most traditional video quality assessment methods without reference are based on statistical analysis, and generally perform statistical analysis on specific video distortion types, which makes the existing methods based on statistical analysis limited in scope of application and poor in real-time performance. Alternatively, the temporal distortion information of the video is characterized based on the characteristics of the block structure similarity between adjacent frames in the video. However, since the noise intensity between frames in the same video may be similar, the structural similarity of corresponding small blocks between adjacent frames is difficult to objectively reflect the quality of the video, resulting in a low accuracy rate in video quality assessment. Summary of the invention

[0003] The embodiments of the present application provide a data quality assessment method, device, computer, and readable storage medium, which can improve the accuracy of quality prediction of multimedia data.

[0004] On the one hand, an embodiment of the present application provides a data quality assessment method, the method comprising:

[0005] Acquire a media feature graph of the multimedia data, and acquire at least two data channels used to constitute the multimedia data;

[0006] Based on the channel attention mechanism, a channel attention mask corresponding to at least two data channels is generated, and a channel feature map of multimedia data is generated according to the channel attention mask and the media feature map;

[0007] Based on the spatial attention mechanism, a spatial attention mask corresponding to the channel feature map is generated, and a spatial feature map of multimedia data is generated according to the spatial attention mask and the channel feature map;

[0008] Output the predicted media quality of multimedia data based on the spatial feature map.

[0009] Wherein, the at least two data channels include data channel i; i is a positive integer, i is the number of data channels less than or equal to the at least two data channels;

[0010] Based on the channel attention mechanism, generate channel attention masks corresponding to at least two data channels, including:

[0011] Based on the media feature graph, determine the data features of the multimedia data in the data channel i;

[0012] Based on the channel attention mechanism, the data features in the data channel i are fused to obtain the channel fusion features of the data channel i;

[0013] When the channel fusion features corresponding to each data channel are obtained, channel attention masks corresponding to at least two data channels are generated according to the channel fusion features corresponding to each data channel.

[0014] Wherein, the channel fusion feature includes the first channel fusion feature and the second channel fusion feature;

[0015] Based on the channel attention mechanism, the data features in the data channel i are fused to obtain the channel fusion features of the data channel i, including:

[0016] Based on the channel attention mechanism, the data features of data channel i are average-pooled to obtain the first channel fusion features of data channel i;

[0017] The mean square error of the data features of data channel i is obtained, and the mean square error is determined as the second channel fusion feature of data channel i.

[0018] Among them, according to the channel fusion features corresponding to each data channel, a channel attention mask corresponding to at least two data channels is generated, including:

[0019] The first channel fusion features corresponding to each data channel are concatenated to obtain a first mean feature, and the second channel fusion features corresponding to each data channel are concatenated to obtain a first mean square feature;

[0020] Performing weighted processing on the first mean feature to obtain a first mean weighted feature, and performing weighted processing on the first mean square feature to obtain a first mean square weighted feature;

[0021] The first mean weighted feature and the first mean square weighted feature are fused to obtain a channel attention mask corresponding to at least two data channels.

[0022] Wherein, the channel attention mask includes a sub-mask for each data channel;

[0023] According to the channel attention mask and the media feature map, a channel feature map of multimedia data is generated, including:

[0024] Based on the sub-mask of data channel i, mask weighting is performed on the data features in data channel i to obtain the channel mask features of data channel i;

[0025] When the channel mask features corresponding to each data channel are obtained, the channel mask features corresponding to each data channel are feature spliced ​​to obtain a channel feature map of the multimedia data.

[0026] Among them, based on the spatial attention mechanism, the spatial attention mask corresponding to the channel feature map is generated, including:

[0027] Obtain at least two pixel points for constituting a channel feature map; the at least two pixel points include pixel point j, j is a positive integer, and j is less than or equal to the number of pixel points of the at least two pixel points;

[0028] Based on the media feature map, determine the channel pixel features corresponding to the pixel point j in each data channel;

[0029] Based on the spatial attention mechanism, feature fusion is performed on at least two channel pixel features corresponding to pixel point j to obtain the pixel fusion feature of pixel point j;

[0030] When the pixel fusion features corresponding to each pixel are obtained, a spatial attention mask corresponding to the channel feature map is generated according to the pixel fusion features corresponding to each pixel.

[0031] Wherein, the pixel fusion feature includes a first pixel fusion feature and a second pixel fusion feature;

[0032] Based on the spatial attention mechanism, feature fusion is performed on at least two channel pixel features corresponding to pixel point j to obtain the pixel fusion feature of pixel point j, including:

[0033] Based on the spatial attention mechanism, average pooling is performed on at least two channel pixel features corresponding to pixel j to obtain the first pixel fusion feature of pixel j;

[0034] The mean square error of at least two channel pixel features corresponding to the pixel point j is obtained, and the mean square error corresponding to the pixel point j is determined as the second pixel fusion feature of the pixel point j.

[0035] Among them, according to the pixel fusion features corresponding to each pixel point, the spatial attention mask corresponding to the channel feature map is generated, including:

[0036] Perform feature splicing on the first pixel fusion features corresponding to at least two pixel points to obtain a second mean feature, and perform feature splicing on the second pixel fusion features corresponding to at least two pixel points to obtain a second mean square feature;

[0037] Perform feature splicing on the second mean feature and the second mean square feature to obtain a pixel splicing feature;

[0038] Convolution is performed on the pixel splicing features to generate the spatial attention mask corresponding to the channel feature map.

[0039] Among them, the pixel splicing features are convolved to generate the spatial attention mask corresponding to the channel feature map, including:

[0040] Obtaining data size and convolution parameters of multimedia data, and determining convolution feature size;

[0041] Add a default feature value to the pixel splicing feature to obtain the pixel feature to be convolved; the size of the pixel feature to be convolved is the convolution feature size;

[0042] Convolve the convolution pixel features to generate the spatial attention mask corresponding to the channel feature map.

[0043] Among them, the convolution parameters include the dilation coefficient;

[0044] Convolve the pixel splicing features to generate the spatial attention mask corresponding to the channel feature map, including:

[0045] The kth convolution position corresponding to the convolution kernel in the pixel splicing feature is determined based on the dilation coefficient, and the element at the kth convolution position in the pixel splicing feature is convolved with the convolution kernel to obtain the kth convolution element; k is a positive integer;

[0046] Obtain the convolution step size, determine the (k+1)th convolution position corresponding to the convolution kernel in the pixel splicing feature based on the convolution step size, the dilation coefficient and the kth convolution position, and use the convolution kernel to convolve the element at the (k+1)th convolution position in the pixel splicing feature to obtain the (k+1)th convolution element;

[0047] When the convolution of the pixel-wise splicing features is completed, a spatial attention mask is generated based on the obtained convolution elements.

[0048] Wherein, the multimedia data includes a first image frame and a second image frame;

[0049] Output the predicted media quality of multimedia data based on the spatial feature map, including:

[0050] Predicting the spatial feature map of the first image frame based on the fully connected layer to obtain a first prediction quality of the first image frame;

[0051] Predicting the spatial feature map of the second image frame based on the fully connected layer to obtain a second prediction quality of the second image frame;

[0052] Obtaining a first evaluation weight of the first image frame and a second evaluation weight of the second image frame, weighting the first prediction quality based on the first evaluation weight to obtain a first weighted prediction value, and weighting the second prediction quality based on the second evaluation weight to obtain a second weighted prediction value;

[0053] The sum of the first weighted prediction value and the second weighted prediction value is determined as the predicted media quality of the multimedia data.

[0054] The method further comprises:

[0055] Obtaining a recommendation threshold, and if the predicted media quality is greater than or equal to the recommendation threshold, recommending and displaying the multimedia data;

[0056] If the predicted media quality is less than the recommended threshold, the user equipment uploading the multimedia data is obtained, and a media quality abnormality message is sent to the user equipment; the media quality abnormality message is used to instruct the user equipment to optimize the media quality of the multimedia data.

[0057] In one aspect, an embodiment of the present application provides a data quality assessment device, the device comprising:

[0058] A channel acquisition module, used to acquire a media feature map of multimedia data, and to acquire at least two data channels for constituting the multimedia data;

[0059] A channel processing module is used to generate channel attention masks corresponding to at least two data channels based on a channel attention mechanism, and generate a channel feature map of multimedia data according to the channel attention masks and the media feature map;

[0060] The spatial processing module is used to generate a spatial attention mask corresponding to the channel feature map based on the spatial attention mechanism, and generate a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map;

[0061] The quality prediction module is used to output the predicted media quality of multimedia data according to the spatial feature map.

[0062] Wherein, the at least two data channels include data channel i; i is a positive integer, i is the number of data channels less than or equal to the at least two data channels;

[0063] In terms of generating channel attention masks corresponding to at least two data channels based on a channel attention mechanism, the channel processing module includes:

[0064] A feature determination unit, used to determine data features of multimedia data in data channel i based on a media feature graph;

[0065] A channel feature fusion unit is used to fuse the data features in the data channel i based on the channel attention mechanism to obtain the channel fusion features of the data channel i;

[0066] The channel mask generating unit is used to generate channel attention masks corresponding to at least two data channels according to the channel fusion features corresponding to each data channel when the channel fusion features corresponding to each data channel are obtained.

[0067] Wherein, the channel fusion feature includes a first channel fusion feature and a second channel fusion feature;

[0068] The channel feature fusion unit includes:

[0069] The first pooling subunit is used to perform mean pooling on the data features of data channel i based on the channel attention mechanism to obtain the first channel fusion features of data channel i;

[0070] The second pooling subunit is used to obtain the mean square error of the data features of the data channel i, and determine the mean square error as the second channel fusion feature of the data channel i.

[0071] Wherein, the channel mask generating unit comprises:

[0072] A first splicing subunit is used to perform feature splicing on the first channel fusion features corresponding to each data channel to obtain a first mean feature, and perform feature splicing on the second channel fusion features corresponding to each data channel to obtain a first mean square feature;

[0073] The second concatenation subunit is used to perform weighted processing on the first mean feature to obtain a first mean weighted feature, and to perform weighted processing on the first mean square feature to obtain a first mean square weighted feature;

[0074] The pooling fusion subunit is used to perform feature fusion on the first mean weighted feature and the first mean square weighted feature to obtain a channel attention mask corresponding to at least two data channels.

[0075] Wherein, the channel attention mask includes a sub-mask for each data channel;

[0076] In terms of generating a channel feature map of multimedia data according to the channel attention mask and the media feature map, the channel processing module includes:

[0077] A feature weighting unit, used for performing mask weighting on data features in data channel i based on the sub-mask of data channel i, so as to obtain a channel mask feature of data channel i;

[0078] The atlas generating unit is used to, when the channel mask features corresponding to each data channel are acquired, perform feature splicing on the channel mask features corresponding to each data channel to obtain a channel feature atlas of the multimedia data.

[0079] Among them, in terms of generating a spatial attention mask corresponding to a channel feature map based on a spatial attention mechanism, the spatial processing module includes:

[0080] A pixel acquisition unit, used to acquire at least two pixel points for constituting a channel feature map; the at least two pixel points include a pixel point j, j is a positive integer, and j is less than or equal to the number of pixel points of the at least two pixel points;

[0081] A feature acquisition unit, used to determine the channel pixel features corresponding to the pixel point j in each data channel based on the media feature map;

[0082] A pixel feature fusion unit is used to fuse the pixel features of at least two channels corresponding to the pixel point j based on the spatial attention mechanism to obtain the pixel fusion feature of the pixel point j;

[0083] The spatial mask generation unit is used to generate a spatial attention mask corresponding to the channel feature map according to the pixel fusion features corresponding to each pixel point when the pixel fusion features corresponding to each pixel point are obtained.

[0084] Wherein, the pixel fusion feature includes a first pixel fusion feature and a second pixel fusion feature;

[0085] The pixel feature fusion unit comprises:

[0086] The third pooling subunit is used to perform mean pooling on at least two channel pixel features corresponding to pixel point j based on the spatial attention mechanism to obtain a first pixel fusion feature of pixel point j;

[0087] The fourth pooling subunit is used to obtain the mean square error of at least two channel pixel features corresponding to the pixel point j, and determine the mean square error corresponding to the pixel point j as the second pixel fusion feature of the pixel point j.

[0088] Wherein, the spatial mask generating unit comprises:

[0089] A third stitching subunit is used to stitch the first pixel fusion features corresponding to at least two pixel points to obtain a second mean feature, and to stitch the second pixel fusion features corresponding to at least two pixel points to obtain a second mean square feature;

[0090] A fourth splicing subunit is used to perform feature splicing on the second mean feature and the second mean square feature to obtain a pixel splicing feature;

[0091] The feature convolution subunit is used to convolve the pixel splicing features to generate the spatial attention mask corresponding to the channel feature map.

[0092] The feature convolution subunit includes:

[0093] A size determination subunit, used to obtain the data size and convolution parameters of the multimedia data and determine the convolution feature size;

[0094] The feature updating subunit is used to add a default feature value to the pixel splicing feature to obtain the pixel feature to be convolved; the size of the pixel feature to be convolved is the convolution feature size;

[0095] Update the convolution subunit to convolve the convolution pixel features to generate the spatial attention mask corresponding to the channel feature map.

[0096] Among them, the convolution parameters include the dilation coefficient;

[0097] The updated convolution subunit includes:

[0098] The first convolution subunit is used to determine the kth convolution position corresponding to the convolution kernel in the pixel splicing feature based on the expansion coefficient, and use the convolution kernel to convolve the element at the kth convolution position in the pixel splicing feature to obtain the kth convolution element; k is a positive integer;

[0099] The second convolution subunit is used to obtain the convolution step size, determine the (k+1)th convolution position corresponding to the convolution kernel in the pixel splicing feature based on the convolution step size, the expansion coefficient and the kth convolution position, and use the convolution kernel to convolve the element at the (k+1)th convolution position in the pixel splicing feature to obtain the (k+1)th convolution element;

[0100] The element combination subunit is used to generate a spatial attention mask based on the convolution elements obtained when the convolution of the pixel splicing features is completed.

[0101] Wherein, the multimedia data includes a first image frame and a second image frame;

[0102] The quality prediction module includes:

[0103] A first prediction unit, configured to predict a spatial feature map of a first image frame based on a fully connected layer to obtain a first prediction quality of the first image frame;

[0104] A second prediction unit, configured to predict a spatial feature map of a second image frame based on a fully connected layer to obtain a second prediction quality of the second image frame;

[0105] A prediction weighting unit, configured to obtain a first evaluation weight of the first image frame and a second evaluation weight of the second image frame, weight the first prediction quality based on the first evaluation weight to obtain a first weighted prediction value, and weight the second prediction quality based on the second evaluation weight to obtain a second weighted prediction value;

[0106] The quality determination unit is used to determine the sum of the first weighted prediction value and the second weighted prediction value as the predicted media quality of the multimedia data.

[0107] The device further comprises:

[0108] A data recommendation module is used to obtain a recommendation threshold, and if the predicted media quality is greater than or equal to the recommendation threshold, the multimedia data is recommended for display;

[0109] The abnormal feedback module is used to obtain the user equipment that uploads the multimedia data if the predicted media quality is less than the recommended threshold, and send a media quality abnormality message to the user equipment; the media quality abnormality message is used to instruct the user equipment to optimize the media quality of the multimedia data.

[0110] On the one hand, an embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;

[0111] The processor is connected to the memory and the input and output interface respectively, wherein the input and output interface is used to receive and output data, the memory is used to store computer programs, and the processor is used to call the computer program to execute the data quality assessment method in one aspect of an embodiment of the present application.

[0112] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the data quality assessment method in one aspect of the embodiment of the present application is executed.

[0113] In one aspect, an embodiment of the present application provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional ways in one aspect of the embodiment of the present application.

[0114] Implementing the embodiments of the present application will have the following beneficial effects:

[0115] In an embodiment of the present application, a computer device obtains a media feature map of multimedia data, obtains at least two data channels used to constitute the multimedia data; based on a channel attention mechanism, generates a channel attention mask corresponding to at least two data channels, and generates a channel feature map of the multimedia data according to the channel attention mask and the media feature map; based on a spatial attention mechanism, generates a spatial attention mask corresponding to the channel feature map, and generates a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map; outputs the predicted media quality of the multimedia data according to the spatial feature map. In the present application, through the channel attention mechanism and the spatial attention mechanism, the network can focus on the data channels constituting the multimedia data during the training process, and increase the attention to the distorted and degraded areas, thereby improving the accuracy of the quality prediction of the multimedia data. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0117] Figure 1 This is a network architecture diagram of data quality assessment provided by an embodiment of the present application;

[0118] Figure 2 This is a schematic diagram of a data quality assessment scenario provided by an embodiment of the present application;

[0119] Figure 3 is a flow chart of a method for data quality assessment provided in an embodiment of the present application;

[0120] Figure 4 This is a schematic diagram of generating a channel characteristic spectrum provided in an embodiment of the present application;

[0121] Figure 5 The present application provides a schematic diagram of a generation model structure of a channel feature map;

[0122] Figure 6 It is a recommended determination scene flow diagram provided by an embodiment of the present application;

[0123] Figure 7 This is a schematic diagram of a spatial feature map generation scenario provided in an embodiment of the present application;

[0124] Figure 8 It is a schematic diagram of a generation model structure of a spatial feature map provided in an embodiment of the present application;

[0125] Fig. 9 It is a schematic diagram of a convolution scenario provided in an embodiment of the present application;

[0126] Fig.10 It is a schematic diagram of a multi-layer dilated convolution structure provided in an embodiment of the present application;

[0127] Fig.11 It is a schematic diagram of the architecture of a quality prediction network provided in an embodiment of the present application;

[0128] Fig.12 is a schematic diagram of a quality prediction result provided in an embodiment of the present application;

[0129] Fig.13 is a schematic diagram of a data quality assessment device provided in an embodiment of the present application;

[0130] Fig.14 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0131] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0132] Among them, in an embodiment of the present application, feature extraction and feature processing can be performed on multimedia data based on technologies such as artificial intelligence and learning. For example, multimedia data can be processed based on a channel attention mechanism to obtain a channel feature map, and the channel feature map can be processed based on a spatial attention mechanism to obtain a spatial feature map. The spatial feature map is predicted to obtain the predicted media quality of the multimedia data, so as to achieve regionalized attention to the multimedia data and divide the importance of each data channel, so that attention can be focused on areas such as distortion or degradation of the multimedia data, thereby improving the accuracy of quality prediction of the multimedia data.

[0133] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence, so as to process the characteristics of multimedia data on each data channel and at each pixel point, and make the processing results as similar as possible to the quality assessment results of human intelligence on multimedia data. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0134] Among them, this application mainly involves machine learning / deep learning (such as channel attention mechanism and spatial attention mechanism, etc.). A quality prediction model can be obtained through learning, and the quality prediction model includes a channel attention module and a spatial attention module, which is used to predict the quality of multimedia data.

[0135] Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). Deep learning is the study of the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Among them, deep learning is a complex machine learning algorithm, and the results achieved in speech and image recognition far exceed those of previous related technologies. Deep learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0136] Furthermore, the data in this application can be stored through cloud storage technology or in the storage space of a computer device. Since a large amount of multimedia data may appear, big data technology can also be used to process the multimedia data in this application.

[0137] Among them, cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0138] The multimedia data in this application is stored through cloud storage technology, which improves the efficiency of data storage and data interaction.

[0139] In the examples of this application, see Figure 1 , Figure 1 1 is a network architecture diagram of a data quality assessment provided in an embodiment of the present application, and the embodiment of the present application can be implemented by a computer device. The network of the data quality assessment may include computer devices and user devices, such as computer device 101, user device 102a, user device 102b, and user device 102c, etc. The computer device can exchange data with each user device, and the computer device can implement the solution in the present application to predict the quality of multimedia data.

[0140] like Figure 1 As shown, the computer device 101 can obtain multimedia data that needs to be quality predicted from any user device, or can obtain multimedia data from the storage space of the computer device 101, and perform quality prediction on the multimedia data. Among them, the computer device 101 can perform quality prediction on the multimedia data after obtaining the multimedia data; the computer device 101 can also cache the multimedia data after obtaining the multimedia data, such as storing the multimedia data in the storage space of the computer device 101, and performing periodic quality prediction on the multimedia data stored in the storage space based on the prediction period; or, the computer device 101 can also perform quality prediction on the multimedia data after obtaining the quality prediction request for the multimedia data, etc., which is not limited here.

[0141] For example, the user device 102a sends a quality prediction request for multimedia data to the computer device 101, and the quality prediction request includes multimedia data. The computer device 101 can perform quality prediction on the multimedia data based on the quality prediction request and send the quality prediction result to the user device 102a. Alternatively, the user device 102b sends multimedia data to the computer device 101, and the computer device 101 can perform quality prediction on the multimedia data and determine whether to recommend the multimedia data based on the quality prediction result. Among them, the computer device 101 can process the features of the multimedia data in each data channel based on the channel attention mechanism to obtain the channel feature map of the multimedia data, so as to focus on the features in each data channel of the multimedia data. Further, the computer device 101 performs feature processing on the channel feature map based on the spatial attention mechanism to obtain the spatial feature map of the multimedia data, so as to focus on the distorted or degraded areas in the multimedia data. Through the channel attention mechanism and the spatial attention mechanism, the degraded or distorted areas of the multimedia data can be focused on to improve the accuracy of the quality prediction of the multimedia data.

[0142] It is understandable that the computer devices mentioned in the embodiments of the present application include but are not limited to terminal devices or servers. In other words, the computer device or user device can be a server or a terminal device, or a system composed of a server and a terminal device. Among them, the terminal device mentioned above can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, augmented reality / virtual reality (AR / VR) devices, helmet displays, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (mobile internet device, MID) with network access capabilities, etc., wherein the client has a display function. Among them, the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0143] Optionally, the data involved in the embodiments of the present application can be stored in a server, or in the memory (ie, storage space) of a computer device, or the data can be stored based on cloud storage technology, without limitation herein.

[0144] For further information, see Figure 2 , Figure 2 Schematic diagram of a data quality assessment scenario provided by an embodiment of the present application. Figure 2 As shown, the quality prediction process of multimedia data in the present application can be implemented by a quality prediction model, and the quality prediction model may include a channel attention module and a spatial attention module. Among them, the computer device obtains multimedia data 201, obtains the media feature map 202 of the multimedia data 201, obtains at least two data channels used to constitute the multimedia data 201, and in the channel attention module 203, the multimedia data 202 is feature processed based on the channel attention mechanism to obtain a channel attention mask 2031 corresponding to at least two data channels, and the channel feature map 204 of the multimedia data 201 is generated according to the channel attention mask 2031 and the media feature map 202. In the spatial attention module 205, the channel feature map 204 is feature processed based on the spatial attention mechanism to obtain a spatial attention mask 2051 corresponding to the channel feature map 204, and the spatial feature map 206 of the multimedia data is generated according to the spatial attention mask 2051 and the media feature map 202. The predicted media quality of the multimedia data is output according to the spatial feature map 206.

[0145] For further information, see Figure 3 , Figure 3 This is a flow chart of a method for data quality assessment provided in an embodiment of the present application. Figure 3 As shown, the data quality assessment process includes the following steps:

[0146] Step S301, obtaining a media feature graph of multimedia data, and obtaining at least two data channels used to constitute the multimedia data.

[0147] In an embodiment of the present application, a computer device obtains a media feature map of multimedia data, and obtains at least two data channels for constituting multimedia data, wherein the data channel refers to a channel constituting a feature of a pixel, for example, the data channel may be a color channel, etc. For example, assuming that the multimedia data is red (red, R), green (green, G), and blue (blue, B) data, the multimedia data may be considered to be composed of three data channels, namely, a red (R) channel, a green (G) channel, and a blue (B) channel; assuming that the multimedia data is cyan (C), magenta (M), yellow (Y), and black (K) data, namely, CMYK data, the multimedia data may be considered to be composed of four data channels, namely, a cyan (C) channel, a magenta (M) channel, a yellow (Y) channel, and a black (K) channel; assuming that the multimedia data is 256 color value data, the multimedia data may be considered to be composed of 256 data channels; assuming that the multimedia data is grayscale data, the multimedia data may be considered to be composed of one data channel, etc., without limitation herein. In other words, the number and channel types of the data channels constituting the multimedia data are determined by the data type of the multimedia data, wherein each data channel represents a dedicated detector.

[0148] The multimedia data may be a single image, such as a Joint Photographic Experts Group (JPEG) format image, a Portable Network Graphics (png) format image, a Bitmap (BMP) format image or a Tag Image File Format (TIFF) format image, etc., or a dynamic image, image combination or video composed of at least two image frames, such as a Graphics Interchange Format (GIF) dynamic image, an Audio Video Interleaved (AVI) format video or a Moving Picture Experts Group 4 (mp4) format video, etc., without limitation. It can be considered that any data that can be split to obtain an image can be considered as multimedia data.

[0149] The computer device obtains a media feature map of multimedia data, which is composed of data features corresponding to at least two data channels constituting the multimedia data. The height of the multimedia data is recorded as H, and the width is recorded as W, that is, the size of the multimedia data is H*W, and the number of channels constituting the at least two data channels of the multimedia data is recorded as C. The computer device processes the multimedia data, extracts the data features corresponding to each data channel of the multimedia data, and forms a media feature map of the multimedia data with the data features corresponding to each data channel. The size of the media feature map is C*H*W, which is used to indicate that the multimedia data includes data features corresponding to C data channels, and each data feature is a feature with a dimension of H*W. The media feature map can be recorded as F, and the size of the media feature map can be expressed as F∈□ C*H*W , to illustrate that the media feature map F is a feature with a dimension of C*H*W. If the multimedia data is composed of at least two image frames, the computer device can split the multimedia data into at least two image frames when acquiring the multimedia data, acquire the media feature map of each image frame, obtain at least two media feature maps, and process each media feature map separately.

[0150] Further optionally, when the quality prediction opportunity of multimedia data is reached, the computer device executes this step, obtains the media feature map of the multimedia data, and obtains at least two data channels used to constitute the multimedia data. Wherein, the quality prediction opportunity can be determined based on the prediction period. When a new prediction period starts, it can be considered that the quality prediction opportunity of the multimedia data is reached. For example, the prediction period is one day, and the last time the quality prediction of historical multimedia data was performed was 10:00 on November 23. Then, when the current system network time is 10:00 on November 24, it is determined that the quality prediction opportunity of the multimedia data is reached. At this time, the computer device can obtain the multimedia data and perform quality prediction on the multimedia data, wherein the number of the multimedia data can be one or at least two; wherein, through this method, after the computer device obtains the multimedia data, the multimedia data is stored. When the quality prediction opportunity is reached, the multimedia data is obtained from the storage location of the multimedia data, and the quality prediction is performed on the multimedia data. Alternatively, the quality prediction opportunity can be the time when the computer device obtains the multimedia data. In other words, when the computer device obtains the multimedia data, it is determined that the quality prediction opportunity of the multimedia data is reached, and the quality prediction is performed on the multimedia data. Alternatively, the quality prediction timing may be the time of obtaining the quality prediction request. In other words, when the computer device obtains the quality prediction request for multimedia data, it determines that the quality prediction timing of the multimedia data has arrived and performs quality prediction on the multimedia data.

[0151] Step S302: Based on the channel attention mechanism, generate channel attention masks corresponding to at least two data channels, and generate a channel feature map of multimedia data according to the channel attention masks and the media feature map.

[0152] In an embodiment of the present application, the computer device can perform feature fusion on the features of each pixel in a data channel based on the channel attention mechanism to obtain the channel fusion features of the data channel, until the channel fusion features of each data channel are obtained, and the channel attention masks corresponding to at least two data channels are generated according to the channel fusion features of each data channel. The feature fusion process in each data channel can be executed simultaneously or separately, and the execution order is not limited. The channel attention mask can be denoted as Mc, and the size of the channel attention mask is C*1*1, denoted as Mc∈□ C*1*1 This is because the dimension reduction operation is performed on the data features in each data channel, so that the size of the data features in the data channel is reduced from H*W to 1*1, while the dimension reduction operation is not performed on the data channel in the media feature map, that is, the dimension of the channel attention mask on the data channel is still C, so the size of the channel attention mask is C*1*1.

[0153] Specifically, the at least two data channels include data channel i; i is a positive integer, and i is the number of data channels that is less than or equal to at least two data channels. The computer device can determine the data features of the multimedia data in the data channel i based on the media feature map; and perform feature fusion on the data features in the data channel i based on the channel attention mechanism to obtain the channel fusion features of the data channel i. When the channel fusion features corresponding to each data channel are obtained, the computer device can generate a channel attention mask corresponding to at least two data channels according to the channel fusion features corresponding to each data channel. The computer device can generate a channel feature map of the multimedia data according to the channel attention mask Mc and the media feature map F, wherein the channel feature map is denoted as F'. Since the size of the channel attention mask Mc is C*1*1 and the size of the media feature map F is C*H*W, the size of the channel feature map F' is C*H*W, denoted as F′∈□ C*H*W .

[0154] For example, see Figure 4 , Figure 4 Schematic diagram of generating a channel characteristic spectrum provided by an embodiment of the present application. Figure 4As shown, assuming that the multimedia data is composed of three data channels, the three data channels include data channel 1, data channel 2 and data channel 3, and the dimension of the media feature map of the multimedia data is 3*H*W, wherein the computer device can obtain data features 4011 of data channel 1, data features 4012 of data channel 2 and data features 4013 of data channel 3 based on the media feature map 401, wherein the dimensions of the above three data features are all H*W. In other words, the media feature map 401 of the multimedia data is composed of three data features with a dimension of H*W. The computer device can perform feature fusion on the data feature 4011 of data channel 1 based on the channel attention mechanism to perform a dimension reduction operation on the data feature 4011 to obtain a channel fusion feature 4021 of data channel 1, and the dimension of the channel fusion feature 4021 is 1*1; perform feature fusion on the data feature 4012 of data channel 2 to perform a dimension reduction operation on the data feature 4012 to obtain a channel fusion feature 4022 of data channel 2, and the dimension of the channel fusion feature 4022 is 1*1; perform feature fusion on the data feature 4013 of data channel 3 to perform a dimension reduction operation on the data feature 4013 to obtain a channel fusion feature 4023 of data channel 3, and the dimension of the channel fusion feature 4023 is 1*1. The channel fusion feature 4021 of data channel 1, the channel fusion feature 4022 of data channel 2, and the channel fusion feature 4023 of data channel 3 are processed to obtain a channel attention mask 403 with a dimension of 3*1*1. According to the channel attention mask 403 and the media feature map 401, a channel feature map 404 of the multimedia data is generated.

[0155] The channel fusion feature includes a first channel fusion feature and a second channel fusion feature. When the computer device performs feature fusion on the data features in the data channel i based on the channel attention mechanism to obtain the channel fusion feature of the data channel i, specifically, the data features of the data channel i are averaged and pooled based on the channel attention mechanism to obtain the first channel fusion feature of the data channel i; the mean square error of the data features of the data channel i is obtained, and the mean square error is determined as the second channel fusion feature of the data channel i.

[0156] Further, the computer device may perform feature concatenation on the first channel fusion features corresponding to each data channel to obtain a first mean feature, perform feature concatenation on the second channel fusion features corresponding to each data channel to obtain a first mean square feature, perform weighted processing on the first mean feature to obtain a first mean weighted feature, and perform weighted processing on the first mean square feature to obtain a first mean square weighted feature. The computer device may perform feature fusion on the first mean weighted feature and the first mean square weighted feature to obtain a channel attention mask corresponding to at least two data channels.

[0157] Among them, see Figure 5 , Figure 5 Schematic diagram of a generation model structure of a channel feature map provided in an embodiment of the present application. Figure 5 As shown, the computer device obtains the media feature map 501 of multimedia data, and performs mean pooling on the data features of each data channel in the media feature map 501 (i.e., the data features of data channel 1 are mean pooled, the data features of data channel 2 are mean pooled, ..., the data features of data channel C are mean pooled), and the first channel fusion features corresponding to each data channel are obtained. The first channel fusion features corresponding to each data channel are feature spliced ​​to obtain a first mean feature 5021, and the first mean feature 5021 is a C*1*1 feature. Among them, the feature splicing can be direct splicing, or it can be weighted splicing of the first channel fusion features corresponding to each data channel, which is not limited here. The first mean feature 5021 is weighted based on the first single-layer neural network 5031 to obtain a first mean weighted feature 5041, wherein the first single-layer neural network 5031 may include C neurons, and the C neurons are equivalent to the weights for the first mean feature 5021. In other words, the C first channel fusion features constituting the first mean feature 5021 correspond one-to-one to the C neurons in the first single-layer neural network 5031.

[0158] Similarly, the computer device obtains the media feature map 501 of the multimedia data, obtains the mean square error of the data features of each data channel in the media feature map 501 (i.e., obtains the mean square error of the data features of data channel 1, obtains the mean square error of the data features of data channel 2, ..., obtains the mean square error of the data features of data channel C), and obtains the second channel fusion features corresponding to each data channel. Taking data channel 1 as an example, the computer device can obtain the mean of the data features of data channel 1, obtain the sum of squares of the deviations of the features of each pixel point included in the data features of data channel 1 from the mean, and then divide it by the total number of pixels included in the multimedia data to obtain the average, and determine the average as the second channel fusion feature of data channel 1. Similarly, obtain the second channel fusion features corresponding to other data channels. The second channel fusion features corresponding to each data channel are feature spliced ​​to obtain the first mean square feature 5022, which is the feature of C*1*1. Among them, the feature splicing can be direct splicing, or it can be weighted splicing of the second channel fusion features corresponding to each data channel, which is not limited here. Based on the second single-layer neural network 5032, the first mean square feature 5022 is weighted to obtain the first mean square weighted feature 5042, wherein the second single-layer neural network 5032 can include C neurons, and the C neurons are equivalent to the weights for the first mean square feature 5022. In other words, the C second channel fusion features that constitute the first mean square feature 5022 correspond one-to-one to the C neurons in the second single-layer neural network 5032.

[0159] Further, the computer device performs feature fusion on the first mean weighted feature 5041 and the first mean square weighted feature 5042, and then processes the feature fusion result based on the activation function to obtain a channel attention mask 507 corresponding to at least two data channels. For example, the first mean weighted feature 5041 and the first mean square weighted feature 5042 are feature fused by adding the features indicated by the symbol 505 (⊕), and the feature fusion result is processed based on the activation function 506. The activation function 506 can be an S-shaped growth curve (sigmoid function) or a rectified linear unit (The Rectified Linear Unit, Relu) activation function, etc., which is not limited here. Further, the computer device can generate a channel feature map 508 of multimedia data based on the channel attention mask 507 and the media feature map 501.

[0160] Furthermore, the channel attention mask includes a sub-mask for each data channel. When the computer device generates a channel feature map of multimedia data based on the channel attention mask and the media feature map, the computer device can perform mask weighting on the data features in data channel i based on the sub-mask of data channel i to obtain the channel mask features of data channel i; when the channel mask features corresponding to each data channel are obtained, the channel mask features corresponding to each data channel are feature spliced ​​to obtain the channel feature map of multimedia data. For example, the multimedia data includes three data channels. Assuming that the channel attention mask includes sub-mask "0.2" of data channel 1, sub-mask "0.4" of data channel 2 and sub-mask "0.4" of data channel 3, the data feature F1 of data channel 1 is mask-weighted based on the sub-mask of data channel 1 to obtain the channel mask feature of data channel 1; the data feature F2 of data channel 2 is mask-weighted based on the sub-mask of data channel 2 to obtain the channel mask feature of data channel 2; the data feature F3 of data channel 3 is mask-weighted based on the sub-mask of data channel 3 to obtain the channel mask feature of data channel 3, and the channel mask features of data channel 1, the channel mask features of data channel 2 and the channel mask features of data channel 3 are feature spliced ​​to obtain the channel feature map of the multimedia data, then the channel feature map is F′=0.2F1++0.4F2++0.4F3, where "++" is used to represent feature splicing.

[0161] Among them, Figure 5 As shown, the mean pooling process for the data features of data channel 1 and the mean square error acquisition process for the data features of data channel 1 can be performed independently, so that the training and learning processes of the two do not interfere with each other, which can improve the mining ability of the channel attention module and make the learning effect of the channel attention module better. Through mean pooling, the prediction error caused by the increase in the variance of the estimated value caused by the domain size of the multimedia data can be reduced, and more information of the multimedia data can be retained; by obtaining the mean square error, the deviation between the multimedia data and the mean is determined to reflect the feature changes of the multimedia data and improve the accuracy of the channel attention mask for the multimedia data.

[0162] Step S303: Based on the spatial attention mechanism, a spatial attention mask corresponding to the channel feature map is generated, and a spatial feature map of the multimedia data is generated according to the spatial attention mask and the channel feature map.

[0163] In an embodiment of the present application, a computer device can fuse the features of a pixel in each data channel of multimedia data based on a spatial attention mechanism to obtain the pixel fusion feature of the pixel. Based on this process, the pixel fusion features corresponding to each pixel are obtained. According to the pixel fusion features corresponding to each pixel, a spatial attention mask corresponding to the channel feature map is generated. The feature fusion process of each pixel can be executed simultaneously or separately, and the execution order is not limited. If the size of the channel feature image is C*H*W, the spatial attention mask is denoted as Ms, and the size of the spatial attention mask is 1*H*W, denoted as Ms∈□ 1*H*W This is because the dimension reduction operation is performed on the channel pixel features of each pixel, so that the dimension of the channel pixel features of the pixel is reduced from C to 1, while the dimension reduction operation is not performed between pixels, that is, the dimension of the spatial attention mask on the pixel is still H*W. Therefore, the size of the spatial attention mask is 1*H*W.

[0164] Specifically, the computer device can obtain at least two pixels used to form a channel feature map, and the at least two pixels include pixel j, where j is a positive integer and j is less than or equal to the number of pixels of at least two pixels. Based on the media feature map, the channel pixel features corresponding to pixel j in each data channel are determined; based on the spatial attention mechanism, the at least two channel pixel features corresponding to pixel j are feature fused to obtain the pixel fusion feature of pixel j. When the pixel fusion features corresponding to each pixel are obtained, a spatial attention mask corresponding to the channel feature map is generated according to the pixel fusion features corresponding to each pixel. Further, the computer device can generate a spatial feature map of multimedia data according to the spatial attention mask Ms and the channel feature map F', wherein the spatial feature map is denoted as F", and since the size of the spatial attention mask Ms is 1*H*W and the size of the channel feature map F' is C*H*W, the size of the spatial feature map F" is C*H*W, denoted as F"∈□ C*H*W .

[0165] The pixel fusion feature includes a first pixel fusion feature and a second pixel fusion feature. When the computer device performs feature fusion on at least two channel pixel features corresponding to pixel point j based on the spatial attention mechanism to obtain the pixel fusion feature of pixel point j, specifically, based on the spatial attention mechanism, the at least two channel pixel features corresponding to pixel point j are averaged and pooled to obtain the first pixel fusion feature of pixel point j; the mean square error of at least two channel pixel features corresponding to pixel point j is obtained, and the mean square error corresponding to pixel point j is determined as the second pixel fusion feature of pixel point j.

[0166] Furthermore, the computer device may perform feature stitching on the first pixel fusion features corresponding to at least two pixels to obtain a second mean feature, and perform feature stitching on the second pixel fusion features corresponding to at least two pixels to obtain a second mean square feature. The computer device may perform feature stitching on the second mean feature and the second mean square feature to obtain a pixel stitching feature; and perform convolution on the pixel stitching feature to generate a spatial attention mask corresponding to the channel feature map. Optionally, the computer device may also perform weighted processing on the second mean feature to obtain a second mean weighted feature, perform weighted processing on the second mean square feature to obtain a second mean square weighted feature, and perform feature stitching on the second mean weighted feature and the second mean square feature to obtain a pixel stitching feature.

[0167] Furthermore, the spatial attention mask includes a sub-mask for each pixel. When the computer device generates a spatial feature map of multimedia data based on the spatial attention mask and the channel feature map, the channel pixel feature of pixel j can be mask-weighted based on the sub-mask of pixel j to obtain the pixel mask feature of pixel j. When the pixel mask features corresponding to each pixel are obtained, the pixel mask features corresponding to each pixel are feature spliced ​​to obtain the spatial feature map of multimedia data. For example, the size of the multimedia data is 100*50, and the multimedia data is composed of C data channels. The obtained spatial attention mask can be considered as a 100*50 matrix, where the size of the channel feature map is C*100*50, that is, there are C data channels, and each data channel corresponds to a 100*50 feature matrix. The 100*50 feature matrix in each data channel is dot-multiplied with the spatial attention mask, and C 100*50 weighted feature matrices can be obtained. These C 100*50 weighted feature matrices constitute the spatial feature map of the multimedia data.

[0168] Step S304: output the predicted media quality of the multimedia data according to the spatial feature map.

[0169] In an embodiment of the present application, if the multimedia data is a single image, the multimedia data is predicted based on the fully connected layer to obtain the predicted media quality of the multimedia data. If the multimedia data is a dynamic image, image combination, or video composed of at least two image frames, at least two image frames constituting the multimedia data are obtained, and the predicted qualities corresponding to the at least two image frames are obtained based on the fully connected layer, and the predicted media quality of the multimedia data is determined according to the predicted qualities corresponding to the at least two image frames. Among them, when determining the predicted media quality of the multimedia data according to the predicted qualities corresponding to at least two image frames, the computer device can determine the average of the predicted qualities corresponding to at least two image frames as the predicted media quality of the multimedia data; the computer device can also obtain the weight corresponding to each image frame, and perform weighted summation of the predicted qualities corresponding to each image frame based on the weight to obtain the predicted media quality of the multimedia data; or, a key image frame among the at least two image frames can be obtained, and the predicted qualities of the key image frames can be weighted summed based on the key frame weights to obtain the key quality, and the image frames other than the key image frames can be weighted summed based on the regular frame weights to obtain the regular quality, and the sum of the key quality and the regular quality can be determined as the predicted media quality of the multimedia data, etc.

[0170] Among them, taking the multimedia data including the first image frame and the second image frame as an example, the computer device can predict the spatial feature map of the first image frame based on the fully connected layer to obtain the first predicted quality of the first image frame; and predict the spatial feature map of the second image frame based on the fully connected layer to obtain the second predicted quality of the second image frame. Further, the computer device can obtain the first evaluation weight of the first image frame and the second evaluation weight of the second image frame, weight the first prediction quality based on the first evaluation weight to obtain the first weighted prediction value, and weight the second prediction quality based on the second evaluation weight to obtain the second weighted prediction value; the sum of the first weighted prediction value and the second weighted prediction value is determined as the predicted media quality of the multimedia data. Optionally, the first frame position of the first image frame in the multimedia data can be obtained. If the first frame position belongs to the first position range, the first frame weight is determined as the weight of the first image frame; if the first frame position belongs to the second position range, the second frame weight is determined as the first evaluation weight. The second frame position of the second image frame in the multimedia data is obtained. If the second frame position belongs to the first position range, the first frame weight is determined as the weight of the second image frame; if the second frame position belongs to the second position range, the second frame weight is determined as the second evaluation weight. The first prediction quality is weighted by the first evaluation weight to obtain a first weighted prediction value, and the second prediction quality is weighted by the second evaluation weight to obtain a second weighted prediction value. The sum of the first weighted prediction value and the second weighted prediction value is determined as the predicted media quality of the multimedia data. The first position range may be the position of a more critical image frame, and the second position range may be the position of a more conventional image frame, that is, the importance of the first position range is greater than the second position range, and the first frame weight is greater than the second frame weight. For example, assuming that the multimedia data is a video, if the importance of the cover frame is considered to be greater than the content frame, the cover frame can be considered to be the first position range, and the content frame can be considered to be the second position range. The first position range and the second position range can be changed based on needs, and are not limited here.

[0171] Optionally, if the multimedia data is a single image, the computer device may perform feature fusion of the media feature map and the spatial feature map to obtain an output feature map, and predict the output feature map based on the fully connected layer to obtain the predicted media quality of the multimedia data. If the multimedia data is a dynamic image, image combination, or video composed of at least two image frames, when obtaining the predicted qualities corresponding to at least two image frames based on the fully connected layer, taking image frame p as an example, the media feature map and the spatial feature map corresponding to image frame p may be feature fused to obtain the output feature map of image frame p, and the output feature map of image frame p may be predicted based on the fully connected layer to obtain the predicted quality of image frame p, where p is a positive integer. Similarly, the predicted quality corresponding to each image frame is obtained.

[0172] Further, the computer device can obtain a recommendation threshold, and if the predicted media quality is greater than or equal to the recommendation threshold, the multimedia data is recommended for display; if the predicted media quality is less than the recommended threshold, the user device that uploads the multimedia data is obtained, and a media quality abnormality message is sent to the user device, and the media quality abnormality message is used to instruct the user device to optimize the media quality of the multimedia data. Optionally, if the first evaluation weight and the second evaluation weight are determined according to the first position range and the second position range, the recommendation threshold includes a first recommendation threshold and a second recommendation threshold, and the first weighted prediction value is compared with the first recommendation threshold, and the second weighted prediction value is compared with the second recommendation threshold. If the first weighted prediction value is greater than or equal to the first recommendation threshold, and the second weighted prediction value is greater than or equal to the second recommendation threshold, the multimedia data is recommended for display; if the first weighted prediction value is less than the first recommendation threshold, or the second weighted prediction value is less than the second recommendation threshold, the user device that uploads the multimedia data is obtained, and a media quality abnormality message is sent to the user device, and the media quality abnormality message is used to instruct the user device to optimize the media quality of the multimedia data.

[0173] For example, see Figure 6 , Figure 6 This is a recommended scene flow chart provided by the embodiment of the present application. Figure 6 As shown, assuming that the multimedia data is a video, at least two image frames constituting the multimedia data are obtained, and the at least two image frames are divided into a video cover frame and a video content frame. The quality of the video cover frame is predicted based on the quality prediction model to obtain a first predicted quality, and the first predicted quality is weighted by a first evaluation weight to obtain a first weighted prediction value; the quality of the video content frame is predicted based on the quality prediction model to obtain a second predicted quality, and the second predicted quality is weighted by a second evaluation weight to obtain a second weighted prediction value. Based on the first weighted prediction value and the second weighted prediction value, a recommendation result for the multimedia data is determined. Wherein, when the computer device obtains at least two image frames constituting the multimedia data, it can collect at least two image frames to be detected from the at least two image frames based on a sampling period, determine the predicted qualities corresponding to the at least two image frames to be detected based on the quality prediction model, and determine the predicted media quality of the multimedia data according to the predicted qualities corresponding to the at least two image frames to be detected. Among them, the quality prediction model includes a channel attention module, a spatial attention module and a fully connected layer, and the quality prediction model is used to implement steps S301 to S304, wherein the channel attention module is used to implement step S302 to obtain the channel feature map of multimedia data; the spatial attention module is used to implement step S303 to obtain the spatial feature map of multimedia data; the fully connected layer is used to implement step S304 to output the predicted media quality of the multimedia data.

[0174] In an embodiment of the present application, a computer device obtains a media feature map of multimedia data, obtains at least two data channels used to constitute the multimedia data; based on a channel attention mechanism, generates a channel attention mask corresponding to at least two data channels, and generates a channel feature map of the multimedia data according to the channel attention mask and the media feature map; based on a spatial attention mechanism, generates a spatial attention mask corresponding to the channel feature map, and generates a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map; outputs the predicted media quality of the multimedia data according to the spatial feature map. In the present application, through the channel attention mechanism and the spatial attention mechanism, the network can focus on the data channels constituting the multimedia data during the training process, and increase the attention to the distorted and degraded areas, thereby improving the accuracy of the quality prediction of the multimedia data.

[0175] For further information, see Figure 7 , Figure 7 Schematic diagram of a spatial feature map generation scenario provided by an embodiment of the present application. Figure 7 As shown, the method for generating the spatial feature map includes the following steps:

[0176] Step S701, based on the spatial attention mechanism, feature fusion processing is performed on the channel feature map between data channels to obtain pixel splicing features.

[0177] In the embodiment of the present application, the computer device can obtain the second mean feature and the second mean square feature of at least two pixels based on the spatial attention mechanism, and perform feature splicing on the second mean feature and the second mean square feature to obtain a pixel splicing feature. The process can be referred to as Figure 3 For further information, see Figure 8 , Figure 8 Schematic diagram of a generation model structure of a spatial feature map provided in an embodiment of the present application. Figure 8As shown, the computer device obtains the channel feature map 801 of the multimedia data, and performs mean pooling on the channel pixel features of each pixel in the channel feature map 801 (i.e., the channel pixel features of pixel 1 are averaged, the channel pixel features of pixel 2 are averaged, ..., the channel pixel features of pixel H*W are averaged), and the first pixel fusion features corresponding to each pixel are obtained. The first pixel fusion features corresponding to each pixel are feature spliced ​​to obtain a second mean feature 8021, and the second mean feature 8021 is a feature of 1*H*W. Among them, the feature splicing can be direct splicing, or it can be weighted splicing of the first pixel fusion features corresponding to each pixel, which is not limited here. The second mean feature 8021 is weighted based on the third single-layer neural network 8031 ​​to obtain a second mean weighted feature, wherein the third single-layer neural network 8031 ​​may include H*W neurons, and the H*W neurons are equivalent to the weights for the second mean feature 8021. In other words, the H*W first pixel fusion features constituting the second mean feature 8021 correspond one-to-one to the H*W neurons in the third single-layer neural network 8031.

[0178] Similarly, the computer device obtains the channel feature map 801 of the multimedia data, obtains the mean square error of the channel pixel feature of each pixel in the channel feature map 801 (i.e., obtains the mean square error of the channel pixel feature of pixel 1, obtains the mean square error of the channel pixel feature of pixel 2, ..., obtains the mean square error of the channel pixel feature of pixel H*W), and obtains the second pixel fusion feature corresponding to each pixel. Taking pixel 1 as an example, the computer device can obtain the mean of the channel pixel features corresponding to pixel 1 in each data channel, obtain the sum of the squares of the differences of the channel pixel features corresponding to pixel 1 in each data channel from the mean, and then divide it by the total number of data channels included in the multimedia data to obtain the average, and determine the average as the second pixel fusion feature of pixel 1. Similarly, obtain the second pixel fusion features corresponding to other pixels. The second pixel fusion features corresponding to each pixel are feature spliced ​​to obtain the second mean square feature 8022, which is a feature of 1*H*W. Among them, the feature splicing can be direct splicing, or it can be weighted splicing of the second pixel fusion features corresponding to each pixel point, which is not limited here. The second mean square feature 8022 is weighted based on the fourth single-layer neural network 8032 to obtain a second mean square weighted feature, wherein the fourth single-layer neural network 8032 may include H*W neurons, and the H*W neurons are equivalent to the weights for the second mean square feature 8022. In other words, the H*W second pixel fusion features constituting the second mean square feature 8022 correspond one-to-one to the H*W neurons in the fourth single-layer neural network 8032.

[0179] Among them, the third single-layer neural network 8031 ​​and the fourth single-layer neural network 8032 can be convolutional networks.

[0180] Furthermore, the computer device performs feature splicing on the second mean weighted feature and the second mean square weighted feature to obtain a pixel splicing feature 804. At this time, the pixel splicing feature 804 is a 2*H*W feature.

[0181] Step S702, processing the pixel splicing features to generate a spatial feature map.

[0182] In an embodiment of the present application, a computer device may perform convolution on a pixel splicing feature to generate a spatial attention mask corresponding to a channel feature map, and generate a spatial feature map according to the spatial attention mask and the channel feature map. Wherein, the size of the convolution kernel for convolving the pixel splicing feature is determined according to the size of the pixel splicing feature. For example, if the size of the pixel splicing feature is 2*H*W, the size of the convolution kernel may be 2*n*n, where n is a positive integer, and the value of n may be changed as needed, such as n may be 3 or 7. Optionally, the computer device may perform continuous convolution on the pixel splicing feature to generate a spatial attention mask corresponding to a channel feature map; the computer device may also perform dilated convolution on the pixel splicing feature to generate a spatial attention mask corresponding to the channel pixel feature. Wherein, continuous convolution means that when the pixel splicing feature is convolved, the convolved elements are adjacent; dilated convolution means that the elements with element spacing in the pixel splicing feature are convolved, which can improve the receptive field of each element in the spatial attention mask. Among them, in the convolutional neural network, the receptive field refers to the size of the area mapped by the pixel points on the feature map (featuremap) output by each layer in the convolutional neural network on the input matrix or input image. In simple terms, the receptive field refers to the area on the input matrix or input image corresponding to a point on the above feature map. In the embodiment of the present application, the number of elements in the pixel splicing feature fused by an element in the spatial attention mask is the receptive field of the spatial attention mask for the input layer.

[0183] In which, taking the case where a computer device performs dilated convolution on a pixel splicing feature as an example, the convolution parameters include a dilation coefficient, and the dilation coefficient is used to indicate the element spacing between the convolved elements when the pixel splicing feature is convolved. The computer device can determine the kth convolution position corresponding to the convolution kernel in the pixel splicing feature based on the dilation coefficient, and use the convolution kernel to convolve the element at the kth convolution position in the pixel splicing feature to obtain the kth convolution element; k is a positive integer; obtain the convolution step length, and determine the (k+1)th convolution position corresponding to the convolution kernel in the pixel splicing feature based on the convolution step length, the dilation coefficient, and the kth convolution position, and use the convolution kernel to convolve the element at the (k+1)th convolution position in the pixel splicing feature to obtain the (k+1)th convolution element; when the convolution of the pixel splicing feature is completed, generate a spatial attention mask according to the obtained convolution element.

[0184] Optionally, the computer device can obtain the data size and convolution parameters of the multimedia data to determine the convolution feature size; add a default feature value to the pixel splicing feature to obtain the pixel feature to be convolved, so as to realize the convolution without dimensionality reduction of the pixel splicing feature, and the size of the pixel feature to be convolved is the convolution feature size. Convolve the pixel feature to be convolved to generate a spatial attention mask corresponding to the channel feature map. Optionally, the computer device can perform continuous convolution on the pixel feature to be convolved to generate a spatial attention mask corresponding to the channel feature map; the computer device can also perform dilated convolution on the pixel feature to be convolved to generate a spatial attention mask corresponding to the channel pixel feature. Among them, the dilated convolution process of the convolved pixel feature is the same as the process of directly dilated convolution on the pixel splicing feature, which will not be repeated here.

[0185] like Figure 8 As shown, the computer device can convolve the pixel splicing feature 804 to generate a pixel fusion map 805, and process the pixel fusion map 805 based on the activation function 806 to obtain a spatial attention mask 807. Among them, the activation function 806 can be an S-shaped growth curve (sigmoid function) or a rectified linear unit (The Rectified Linear Unit, Relu) activation function, etc., which is not limited here. Further, the computer device can generate a spatial feature map 808 of multimedia data based on the spatial attention mask 807 and the channel feature map 801.

[0186] Among them, see Fig. 9 , Fig. 9 Schematic diagram of a convolution scenario provided by an embodiment of the present application. Fig. 9As shown, assuming that the pixel splicing feature 901 is a 7*8 feature, the data size of the multimedia data is 7*8, and the expansion coefficient included in the convolution parameter is used to indicate that the element interval is 1. In order to perform convolution on the pixel splicing feature 901 without dimensionality reduction, that is, the size of the spatial attention mask obtained after convolution on the pixel splicing feature 901 is 7*8, the computer device can obtain the data size and convolution parameters of the multimedia data, and determine that the convolution feature size is 11*12. A default feature value is added to the pixel splicing feature 901 to obtain the pixel feature 902 to be convolved, wherein the adding position of the default feature value in the pixel splicing feature 901 can be set as needed, such as the adding position is evenly distributed to the four sides of the pixel splicing feature 901, or the adding position can be on a single side of the width or height of the pixel splicing feature 901. For example, assuming that the pixel splicing feature 901 abstracts four sides, which are respectively recorded as A1, B1, A2 and B2, wherein A1 and A2 are relative sides, and B1 and B2 are relative sides, then the default feature value can be added on both sides of A1 and B1; or, the adding position can be the starting position or the ending position of the pixel splicing feature 901, etc., which is not limited here.

[0187] Among them, the size of the pixel feature 902 to be convolved obtained by the computer device is 11*12, and the feature corresponding to the dotted line part is the added default feature value. Further, assuming that the convolution step size of the convolution kernel is 1, the computer device determines the first convolution position corresponding to the convolution kernel in the pixel feature 902 to be convolved based on the expansion coefficient as (the first row first, the first row third, the first row fifth, the third row third, the third row fifth, the fifth row first, the fifth row third and the fifth row fifth), and uses the convolution kernel to convolve the element at the first convolution position in the pixel feature 902 to be convolved to obtain the first convolution element; according to the convolution step size "1", the expansion coefficient and the first convolution position, it is determined that the second convolution position corresponding to the convolution kernel in the pixel splicing feature is (the first row second, the first row fourth, the first row sixth, the third row second, the third row fifth The fourth, the sixth in the third row, the second in the fifth row, the fourth in the fifth row, and the sixth in the fifth row), the convolution kernel is used to convolve the element at the second convolution position in the convolution pixel feature 902 to obtain the second convolution element; ...; according to the convolution step size "1", the expansion coefficient and the 55th convolution position, the last convolution position corresponding to the convolution kernel in the pixel splicing feature is determined to be (the seventh in the eighth row, the ninth in the eighth row, the eleventh in the eighth row, the seventh in the tenth row, the ninth in the tenth row, the eleventh in the tenth row, the seventh in the twelfth row, the ninth in the twelfth row, and the eleventh in the twelfth row), the convolution kernel is used to convolve the element at the last convolution position in the convolution pixel feature 902 to obtain the last convolution element. According to each convolution element obtained, a pixel fusion map 903 is generated, and the pixel fusion map 903 is processed based on the activation function to obtain a spatial attention mask 904.

[0188] Optionally, the computer device may perform multi-layer convolution on the pixel splicing features to obtain a spatial attention mask; or generate a pixel feature to be convolved according to the pixel splicing features, and perform multi-layer convolution on the pixel feature to be convolved to obtain a spatial attention mask. Fig.10 , Fig.10 Schematic diagram of a multi-layer dilated convolution structure provided in an embodiment of the present application. Fig.10As shown, assuming that the number of convolution layers is 3 and the step size is 1, taking the pixel splicing feature as an example, assuming that the pixel splicing feature is (x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11). The expansion coefficient of the first dilated convolution layer is obtained. The expansion coefficient of the first dilated convolution layer is 1, which means that the element interval of the first dilated convolution layer is 0. The computer device convolves x1, x2 and x3 according to the expansion coefficient of the first dilated convolution layer to obtain the first element in the first dilated convolution layer; convolve x2, x3 and x4 to obtain the second element in the first dilated convolution layer; convolve x3, x4 and x5 to obtain the third element in the first dilated convolution layer; ...; convolve x9, x10 and x11 to obtain the ninth element in the first dilated convolution layer. The nine elements in the first dilated convolution layer are spliced ​​to obtain the output matrix of the first dilated convolution layer.

[0189] The computer device determines the expansion coefficient of the second expansion convolution layer according to the expansion coefficient of the first expansion convolution layer, and obtains the expansion coefficient of the second expansion convolution layer as 2. According to the expansion coefficient of the second expansion convolution layer, the element spacing of the second expansion convolution layer is determined to be 1, and the first element, the third element and the fifth element in the output matrix of the first expansion convolution layer are convolved to obtain the first element of the second expansion convolution layer; the second element, the fourth element and the sixth element in the output matrix of the first expansion convolution layer are convolved to obtain the second element of the second expansion convolution layer; ...; the fifth element, the seventh element and the ninth element in the output matrix of the first expansion convolution layer are convolved to obtain the fifth element of the second expansion convolution layer. The five elements of the second expansion convolution layer are concatenated to obtain the output matrix of the second expansion convolution layer. According to the expansion coefficient "2" of the second expansion convolution layer, the expansion coefficient of the third expansion convolution layer is determined, and the expansion coefficient of the third expansion convolution layer is obtained as 4. According to the expansion coefficient of the third expansion convolution layer, the element spacing of the third expansion convolution layer is determined to be 3, and the first element, the fifth element and the ninth element in the output matrix of the second expansion convolution layer are convolved to obtain the first element of the third expansion convolution layer, wherein the output matrix of the second expansion convolution layer can be padded with zeros to convolve the output matrix of the second expansion convolution layer in the third expansion convolution layer. According to the first element of the third expansion convolution layer, the output matrix of the third expansion convolution layer is obtained, and the output matrix of the third expansion convolution layer is determined as the spatial attention mask.

[0190] Through multiple layers of convolution, the expansion coefficient can be increased layer by layer to make the feature processing of the feature convolution of the pixel splicing features or the pixel features to be convolved smoother, and the receptive field of the obtained spatial attention mask can be expanded, so that the spatial attention mask can include more information and improve accuracy.

[0191] In an embodiment of the present application, it is described that the channel feature map is processed based on the spatial attention mechanism, and finally the spatial feature map of the multimedia data is obtained, so that each element in the spatial feature map can carry more information, thereby improving the accuracy of the model.

[0192] Furthermore, the quality prediction model in the embodiment of the present application can be implemented by embedding in a neural network, and the neural network can be a ResNet18 network, a Visual Geometry Group Net (vggnet) or an AlexNet network, etc., wherein the numbers in the ResNet18 network represent the depth of the network, referring to weighted layers, including convolutional layers and fully connected layers, excluding pooling layers and batch normalization layers (BatchNormalization, BN) layers, etc. Among them, the embedding position of the quality prediction model in the neural network can be changed, that is, the quality prediction model can be inserted at any position in the neural network to obtain the final quality prediction network obtained in the embodiment of the present application, or any number of quality prediction models can be inserted in the neural network to obtain the quality prediction network obtained in the embodiment of the present application. In short, the embedding position and embedding quantity of the quality prediction model in the neural network can be adjusted according to actual conditions.

[0193] For example, in a ResNet18 network, the quality prediction model can be embedded between the third convolutional layer and the fourth convolutional layer, or between the fourth convolutional layer and the fifth convolutional layer. Embedding the quality prediction model into the middle layer can make the resolution of the quality prediction model larger, that is, it can extract the global features in the media feature map and obtain some local details, thereby improving the accuracy of quality prediction of multimedia data.

[0194] Among them, see Fig.11 , Fig.11 Schematic diagram of the architecture of a quality prediction network provided by an embodiment of the present application. Fig.11As shown, multimedia data is input into the quality prediction network, and after convolution through multiple convolution blocks, a media feature map 1101 is obtained. In the channel attention module 1102, based on the channel attention mechanism, a channel attention mask 11021 corresponding to the media feature map 1101 in at least two data channels is generated. According to the channel attention mask 11021 and the media feature map 1101, a channel feature map 1103 of the multimedia data is generated. In the spatial attention module 1104, based on the spatial attention mechanism, a spatial attention mask 11041 corresponding to the channel feature map 1103 is generated. According to the spatial attention mask 11041 and the channel feature map 1103, a spatial feature map 1105 of the multimedia data is generated. The computer device can directly determine the spatial feature map 1105 as the output feature map 1106 of the multimedia data; or perform feature fusion on the spatial feature map 1105 and the media feature map 1101 to obtain the output feature map 1106 of the multimedia data. The output feature map 1106 is input into the next convolution block of the quality prediction model, and the output feature map 1106 is convolved until it is input into the last layer (i.e., the fully connected layer) of the quality prediction network, and the acquired features are predicted in the last layer to obtain the predicted quality of the image frame corresponding to the media feature map 1101. If the multimedia data includes a single image, the predicted quality is determined as the predicted media quality of the multimedia data; if the multimedia data includes at least two image frames, the predicted quality corresponding to each image frame is obtained based on the above process, and the predicted media quality of the multimedia data is determined according to the predicted quality corresponding to each image frame.

[0195] It is assumed that the predicted media quality of multimedia data ranges from 0 to 3, and a higher value indicates better quality of the corresponding multimedia data. Fig.12 As shown, Fig.12 This is a schematic diagram of quality prediction results provided by an embodiment of the present application. It can be seen that the predicted media quality of multimedia data 1201 is 0.66, the predicted media quality of multimedia data 1202 is 1.32, and the predicted media quality of multimedia data 1203 is 0.77. It can be seen that the embodiment of the present application can predict a relatively reasonable predicted media quality for multimedia data. For multimedia data with greater compression noise, the value of predicted media quality is smaller. Among them, 848 test samples are used, and the experimental results are as follows:

[0196] The Pearson Line Correlation Coefficient (PLCC) is 0.88, and the Spearman rank order correlation coefficient (SROCC) is 0.87. Based on these two indicators, it can be seen that in the test sample set, the embodiment of the present application has excellent effect in performing reference-free quality prediction on multimedia data.

[0197] For further information, see Fig.13 , Fig.13 Schematic diagram of a data quality assessment device provided in an embodiment of the present application. The data quality assessment device may be a computer program (including program code, etc.) running in a computer device, for example, the data quality assessment device may be an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.13 As shown, the data quality assessment device 1300 can be used to Figure 3 The computer device in the corresponding embodiment may specifically include: a channel acquisition module 11, a channel processing module 12, a space processing module 13 and a quality prediction module.

[0198] A channel acquisition module 11 is used to acquire a media feature map of multimedia data and acquire at least two data channels for constituting the multimedia data;

[0199] A channel processing module 12 is used to generate channel attention masks corresponding to at least two data channels based on a channel attention mechanism, and generate a channel feature map of multimedia data according to the channel attention masks and the media feature map;

[0200] The spatial processing module 13 is used to generate a spatial attention mask corresponding to the channel feature map based on the spatial attention mechanism, and generate a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map;

[0201] The quality prediction module 14 is used to output the predicted media quality of the multimedia data according to the spatial feature map.

[0202] Wherein, the at least two data channels include data channel i; i is a positive integer, i is the number of data channels less than or equal to the at least two data channels;

[0203] In terms of generating channel attention masks corresponding to at least two data channels based on the channel attention mechanism, the channel processing module 12 includes:

[0204] A feature determination unit 121, configured to determine data features of multimedia data in a data channel i based on a media feature graph;

[0205] A channel feature fusion unit 122 is used to perform feature fusion on data features in data channel i based on a channel attention mechanism to obtain channel fusion features of data channel i;

[0206] The channel mask generating unit 123 is used to generate channel attention masks corresponding to at least two data channels according to the channel fusion features corresponding to each data channel when the channel fusion features corresponding to each data channel are acquired.

[0207] Wherein, the channel fusion feature includes a first channel fusion feature and a second channel fusion feature;

[0208] The channel feature fusion unit 122 includes:

[0209] The first pooling subunit 1221 is used to perform mean pooling on the data features of the data channel i based on the channel attention mechanism to obtain the first channel fusion features of the data channel i;

[0210] The second pooling subunit 1222 is used to obtain the mean square error of the data features of the data channel i, and determine the mean square error as the second channel fusion feature of the data channel i.

[0211] The channel mask generating unit 123 includes:

[0212] The first splicing subunit 1231 is used to perform feature splicing on the first channel fusion features corresponding to each data channel to obtain a first mean feature, and perform feature splicing on the second channel fusion features corresponding to each data channel to obtain a first mean square feature;

[0213] The second concatenation subunit 1232 is configured to perform weighted processing on the first mean feature to obtain a first mean weighted feature, and perform weighted processing on the first mean square feature to obtain a first mean square weighted feature;

[0214] The pooling fusion subunit 1233 is used to perform feature fusion on the first mean weighted feature and the first mean square weighted feature to obtain a channel attention mask corresponding to at least two data channels.

[0215] Wherein, the channel attention mask includes a sub-mask for each data channel;

[0216] In terms of generating a channel feature map of multimedia data according to the channel attention mask and the media feature map, the channel processing module 12 includes:

[0217] A feature weighting unit 124 is used to perform mask weighting on the data features in the data channel i based on the sub-mask of the data channel i to obtain a channel mask feature of the data channel i;

[0218] The atlas generating unit 125 is used to perform feature splicing on the channel mask features corresponding to each data channel when the channel mask features corresponding to each data channel are acquired, so as to obtain a channel feature atlas of the multimedia data.

[0219] Among them, in terms of generating a spatial attention mask corresponding to a channel feature map based on a spatial attention mechanism, the spatial processing module 13 includes:

[0220] The pixel acquisition unit 131 is used to acquire at least two pixel points for constituting a channel feature map; the at least two pixel points include a pixel point j, where j is a positive integer, and j is less than or equal to the number of pixel points of the at least two pixel points;

[0221] A feature acquisition unit 132 is used to determine the channel pixel features corresponding to the pixel point j in each data channel based on the media feature map;

[0222] A pixel feature fusion unit 133 is used to perform feature fusion on at least two channel pixel features corresponding to pixel point j based on a spatial attention mechanism to obtain a pixel fusion feature of pixel point j;

[0223] The spatial mask generating unit 134 is used to generate a spatial attention mask corresponding to the channel feature map according to the pixel fusion features corresponding to each pixel point when the pixel fusion features corresponding to each pixel point are obtained.

[0224] Wherein, the pixel fusion feature includes a first pixel fusion feature and a second pixel fusion feature;

[0225] The pixel feature fusion unit 133 includes:

[0226] The third pooling subunit 1331 is used to perform mean pooling on at least two channel pixel features corresponding to the pixel point j based on the spatial attention mechanism to obtain a first pixel fusion feature of the pixel point j;

[0227] The fourth pooling subunit 1332 is used to obtain the mean square error of at least two channel pixel features corresponding to the pixel point j, and determine the mean square error corresponding to the pixel point j as the second pixel fusion feature of the pixel point j.

[0228] The spatial mask generating unit 134 includes:

[0229] The third splicing subunit 1341 is used to perform feature splicing on the first pixel fusion features corresponding to at least two pixel points to obtain a second mean feature, and perform feature splicing on the second pixel fusion features corresponding to at least two pixel points to obtain a second mean square feature;

[0230] The fourth splicing subunit 1342 is used to perform feature splicing on the second mean feature and the second mean square feature to obtain a pixel splicing feature;

[0231] The feature convolution subunit 1343 is used to convolve the pixel splicing features to generate a spatial attention mask corresponding to the channel feature map.

[0232] Among them, the feature convolution subunit 1343 includes:

[0233] The size determination subunit 13431 is used to obtain the data size and convolution parameters of the multimedia data and determine the convolution feature size;

[0234] The feature updating subunit 13432 is used to add a default feature value to the pixel splicing feature to obtain the pixel feature to be convolved; the size of the pixel feature to be convolved is the convolution feature size;

[0235] The convolution subunit 13433 is updated to convolve the convolution pixel features to generate a spatial attention mask corresponding to the channel feature map.

[0236] Among them, the convolution parameters include the dilation coefficient;

[0237] The updated convolution subunit 13433 includes:

[0238] The first convolution subunit 1343a is used to determine the kth convolution position corresponding to the convolution kernel in the pixel splicing feature based on the dilation coefficient, and use the convolution kernel to convolve the element at the kth convolution position in the pixel splicing feature to obtain the kth convolution element; k is a positive integer;

[0239] The second convolution subunit 1343b is used to obtain a convolution step size, determine the (k+1)th convolution position corresponding to the convolution kernel in the pixel splicing feature based on the convolution step size, the dilation coefficient and the kth convolution position, and use the convolution kernel to convolve the element at the (k+1)th convolution position in the pixel splicing feature to obtain the (k+1)th convolution element;

[0240] The element combination subunit 1343c is used to generate a spatial attention mask based on the obtained convolution elements when the pixel splicing feature convolution is completed.

[0241] Wherein, the multimedia data includes a first image frame and a second image frame;

[0242] The quality prediction module 14 includes:

[0243] A first prediction unit 141 is used to predict the spatial feature map of the first image frame based on the fully connected layer to obtain a first prediction quality of the first image frame;

[0244] A second prediction unit 142, configured to predict the spatial feature map of the second image frame based on the fully connected layer to obtain a second prediction quality of the second image frame;

[0245] A prediction weighting unit 143, configured to obtain a first evaluation weight of the first image frame and a second evaluation weight of the second image frame, weight the first prediction quality based on the first evaluation weight to obtain a first weighted prediction value, and weight the second prediction quality based on the second evaluation weight to obtain a second weighted prediction value;

[0246] The quality determination unit 144 is configured to determine the sum of the first weighted prediction value and the second weighted prediction value as the predicted media quality of the multimedia data.

[0247] The device 1300 further includes:

[0248] The data recommendation module 15 is used to obtain a recommendation threshold, and if the predicted media quality is greater than or equal to the recommendation threshold, the multimedia data is recommended for display;

[0249] The abnormal feedback module 16 is used to obtain the user equipment that uploads the multimedia data and send a media quality abnormality message to the user equipment if the predicted media quality is less than the recommended threshold; the media quality abnormality message is used to instruct the user equipment to optimize the media quality of the multimedia data.

[0250] The embodiment of the present application provides a data quality assessment device, which obtains a media feature map of multimedia data, obtains at least two data channels used to constitute the multimedia data; based on a channel attention mechanism, generates a channel attention mask corresponding to at least two data channels, and generates a channel feature map of the multimedia data according to the channel attention mask and the media feature map; based on a spatial attention mechanism, generates a spatial attention mask corresponding to the channel feature map, and generates a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map; outputs the predicted media quality of the multimedia data according to the spatial feature map. In the present application, through the channel attention mechanism and the spatial attention mechanism, the network can focus on the data channels constituting the multimedia data during the training process, and improve the attention to the distorted and degraded areas, thereby improving the accuracy of the quality prediction of the multimedia data.

[0251] See also Fig.14 , Fig.14 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.14As shown, the computer device in the embodiment of the present application may include: one or more processors 1401, a memory 1402, and an input-output interface 1403. The processor 1401, the memory 1402, and the input-output interface 1403 are connected via a bus 1404. The memory 1402 is used to store a computer program, which includes program instructions, and the input-output interface 1403 is used to receive data and output data, such as for data interaction between the computer device and the user device; the processor 1401 is used to execute the program instructions stored in the memory 1402.

[0252] The processor 1401 performs the following operations:

[0253] Acquire a media feature graph of the multimedia data, and acquire at least two data channels used to constitute the multimedia data;

[0254] Based on the channel attention mechanism, a channel attention mask corresponding to at least two data channels is generated, and a channel feature map of multimedia data is generated according to the channel attention mask and the media feature map;

[0255] Based on the spatial attention mechanism, a spatial attention mask corresponding to the channel feature map is generated, and a spatial feature map of multimedia data is generated according to the spatial attention mask and the channel feature map;

[0256] Output the predicted media quality of multimedia data based on the spatial feature map.

[0257] In some feasible implementations, the processor 1401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0258] The memory 1402 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1401 and the input / output interface 1403. A portion of the memory 1402 may also include a non-volatile random access memory. For example, the memory 1402 may also store information about the device type.

[0259] In a specific implementation, the computer device can execute the following operations through its built-in functional modules: Figure 3 For details on the implementation methods provided in each step, please refer to the Figure 3 The implementation methods provided in each step will not be repeated here.

[0260] The embodiment of the present application provides a computer device, including: a processor, an input and output interface, and a memory, wherein the processor obtains computer instructions in the memory and executes the computer instructions. Figure 3 The various steps of the method shown in are used to perform data quality assessment operations. The embodiment of the present application realizes obtaining a media feature map of multimedia data, obtaining at least two data channels used to constitute multimedia data; based on the channel attention mechanism, generating a channel attention mask corresponding to at least two data channels, and generating a channel feature map of multimedia data according to the channel attention mask and the media feature map; based on the spatial attention mechanism, generating a spatial attention mask corresponding to the channel feature map, and generating a spatial feature map of multimedia data according to the spatial attention mask and the channel feature map; outputting the predicted media quality of the multimedia data according to the spatial feature map. In the present application, through the channel attention mechanism and the spatial attention mechanism, the network can focus on the data channels that constitute the multimedia data during the training process, and increase the attention to the distorted and degraded areas, thereby improving the accuracy of the quality prediction of the multimedia data.

[0261] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by the processor, Figure 3 For details on the data quality assessment methods provided in each step, please refer to the Figure 3 The implementation methods provided in each step are not repeated here. In addition, the description of the beneficial effects of adopting the same method is not repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed to be executed on a computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0262] The computer-readable storage medium may be the data quality assessment device provided by any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0263] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 3 The method provided in the various optional methods realizes obtaining the channel feature map corresponding to the multimedia data based on the channel attention mechanism, processing the channel feature map based on the spatial attention mechanism, obtaining the spatial feature map of the multimedia data, determining the predicted media quality of the multimedia data according to the spatial feature map, and combining the channel attention mechanism and the spatial attention mechanism to realize the attention to the distorted and degraded areas in the multimedia data, thereby improving the accuracy of quality prediction. The quality prediction model in this application is based on end-to-end training, which can optimize the quality prediction model in the iterative process, while maintaining the simplicity of the convolutional attention module and improving the prediction speed.

[0264] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0265] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0266] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0267] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data quality assessment method, characterized in that: The method comprises: Acquire a media feature graph of multimedia data, where the multimedia data includes a first image frame and a second image frame; Acquire at least two data channels for constituting the multimedia data; Based on the channel attention mechanism, generate a channel attention mask corresponding to the at least two data channels, and generate a channel feature map of the multimedia data according to the channel attention mask and the media feature map; Based on the spatial attention mechanism, the channel feature map is subjected to feature fusion processing between data channels to obtain pixel splicing features; Perform at least two layers of dilated convolution on the pixel splicing features to generate a spatial attention mask corresponding to the channel feature map, where each dilated convolution layer corresponds to a dilation coefficient and the dilation coefficient of the subsequent dilated convolution layer is greater than the dilation coefficient of the previous dilated convolution layer; Generating a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map; Predicting the spatial feature map of the first image frame based on the fully connected layer to obtain a first prediction quality of the first image frame; Predicting the spatial feature map of the second image frame based on the fully connected layer to obtain a second prediction quality of the second image frame; Obtaining a first evaluation weight of the first image frame and a second evaluation weight of the second image frame, weighting the first prediction quality based on the first evaluation weight to obtain a first weighted prediction value, and weighting the second prediction quality based on the second evaluation weight to obtain a second weighted prediction value; The sum of the first weighted prediction value and the second weighted prediction value is determined as the predicted media quality of the multimedia data.

2. The method according to claim 1, characterized in that The at least two data channels include data channel i; i is a positive integer, i is the number of data channels less than or equal to the at least two data channels; The generating, based on the channel attention mechanism, a channel attention mask corresponding to the at least two data channels comprises: Determining data features of the multimedia data in the data channel i based on the media feature graph; Based on the channel attention mechanism, feature fusion is performed on the data features in the data channel i to obtain the channel fusion features of the data channel i; When the channel fusion features respectively corresponding to each data channel are obtained, the channel attention masks corresponding to the at least two data channels are generated according to the channel fusion features respectively corresponding to each data channel.

3. The method according to claim 2, characterized in that The channel fusion feature includes a first channel fusion feature and a second channel fusion feature; The step of performing feature fusion on the data features in the data channel i based on the channel attention mechanism to obtain the channel fusion features of the data channel i includes: Perform mean pooling on the data features of the data channel i based on the channel attention mechanism to obtain the first channel fusion features of the data channel i; A mean square error of the data feature of the data channel i is obtained, and the mean square error is determined as the second channel fusion feature of the data channel i.

4. The method according to claim 3, characterized in that The generating, according to the channel fusion features respectively corresponding to each of the data channels, the channel attention masks corresponding to the at least two data channels comprises: Perform feature splicing on the first channel fusion features corresponding to each data channel to obtain a first mean feature, and perform feature splicing on the second channel fusion features corresponding to each data channel to obtain a first mean square feature; Performing weighted processing on the first mean feature to obtain a first mean weighted feature, and performing weighted processing on the first mean square feature to obtain a first mean square weighted feature; The first mean weighted feature and the first mean square weighted feature are feature fused to obtain a channel attention mask corresponding to the at least two data channels.

5. The method according to claim 2, characterized in that The channel attention mask comprises a sub-mask for each data channel; The step of generating a channel feature map of the multimedia data according to the channel attention mask and the media feature map includes: Based on the sub-mask of the data channel i, mask weighting is performed on the data features in the data channel i to obtain the channel mask features of the data channel i; When the channel mask features respectively corresponding to each data channel are obtained, the channel mask features respectively corresponding to each data channel are feature spliced ​​to obtain a channel feature map of the multimedia data.

6. The method according to claim 1, characterized in that The generating of the spatial attention mask corresponding to the channel feature map based on the spatial attention mechanism includes: Acquire at least two pixel points for constituting the channel feature map; the at least two pixel points include pixel point j, j is a positive integer, and j is less than or equal to the number of pixel points of the at least two pixel points; Based on the media feature map, determining the channel pixel features corresponding to the pixel point j in each data channel; Based on the spatial attention mechanism, feature fusion is performed on at least two channel pixel features corresponding to the pixel point j to obtain a pixel fusion feature of the pixel point j; When the pixel fusion features corresponding to each pixel point are obtained, a spatial attention mask corresponding to the channel feature map is generated according to the pixel fusion features corresponding to each pixel point.

7. The method according to claim 6, characterized in that The pixel fusion feature includes a first pixel fusion feature and a second pixel fusion feature; The method of performing feature fusion on at least two channel pixel features corresponding to the pixel point j based on the spatial attention mechanism to obtain the pixel fusion feature of the pixel point j includes: Based on the spatial attention mechanism, mean pooling is performed on at least two channel pixel features corresponding to the pixel point j to obtain a first pixel fusion feature of the pixel point j; The mean square error of at least two channel pixel features corresponding to the pixel point j is obtained, and the mean square error corresponding to the pixel point j is determined as the second pixel fusion feature of the pixel point j.

8. The method according to claim 7, characterized in that The step of generating a spatial attention mask corresponding to the channel feature map according to the pixel fusion features corresponding to each pixel point includes: Perform feature splicing on the first pixel fusion features respectively corresponding to the at least two pixel points to obtain a second mean feature, and perform feature splicing on the second pixel fusion features respectively corresponding to the at least two pixel points to obtain a second mean square feature; Perform feature splicing on the second mean feature and the second mean square feature to obtain a pixel splicing feature; The pixel splicing features are convolved to generate a spatial attention mask corresponding to the channel feature map.

9. The method according to claim 8, characterized in that The convolving the pixel splicing features to generate a spatial attention mask corresponding to the channel feature map includes: Obtaining the data size and convolution parameters of the multimedia data, and determining the convolution feature size; Adding a default feature value to the pixel splicing feature to obtain a pixel feature to be convolved; the size of the pixel feature to be convolved is the convolution feature size; Convolve the pixel features to be convolved to generate a spatial attention mask corresponding to the channel feature map.

10. The method according to claim 8, characterized in that The convolution parameters include dilation coefficients; The convolving the pixel splicing features to generate a spatial attention mask corresponding to the channel feature map includes: Determine the kth convolution position corresponding to the convolution kernel in the pixel splicing feature based on the dilation coefficient, and use the convolution kernel to convolve the element at the kth convolution position in the pixel splicing feature to obtain the kth convolution element; k is a positive integer; Obtaining a convolution step size, determining a (k+1)th convolution position corresponding to the convolution kernel in the pixel splicing feature based on the convolution step size, the dilation coefficient, and the kth convolution position, and performing convolution on the element at the (k+1)th convolution position in the pixel splicing feature using the convolution kernel to obtain a (k+1)th convolution element; When the convolution of the pixel splicing features is completed, a spatial attention mask is generated according to the obtained convolution elements.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: Obtaining a recommendation threshold, and if the predicted media quality is greater than or equal to the recommendation threshold, recommending and displaying the multimedia data; If the predicted media quality is less than the recommended threshold, a user equipment that uploads the multimedia data is obtained, and a media quality abnormality message is sent to the user equipment; the media quality abnormality message is used to instruct the user equipment to optimize the media quality of the multimedia data.

12. A data quality assessment device, characterized in that: The device comprises: A channel acquisition module, used to acquire a media feature map of multimedia data, and acquire at least two data channels used to constitute the multimedia data, wherein the multimedia data includes a first image frame and a second image frame; A channel processing module, configured to generate a channel attention mask corresponding to the at least two data channels based on a channel attention mechanism, and generate a channel feature map of the multimedia data according to the channel attention mask and the media feature map; A spatial processing module is used to perform feature fusion processing between data channels on the channel feature map based on a spatial attention mechanism to obtain a pixel splicing feature; perform at least two layers of dilated convolution on the pixel splicing feature to generate a spatial attention mask corresponding to the channel feature map, and generate a spatial feature map of the multimedia data according to the spatial attention mask and the channel feature map, wherein each dilated convolution layer corresponds to a dilation coefficient and the dilation coefficient of the subsequent dilated convolution layer is greater than the dilation coefficient of the previous dilated convolution layer; A quality prediction module, configured to output a predicted media quality of the multimedia data according to the spatial feature map; The quality prediction module comprises: A first prediction unit, configured to predict a spatial feature map of a first image frame based on a fully connected layer to obtain a first prediction quality of the first image frame; A second prediction unit, configured to predict a spatial feature map of a second image frame based on a fully connected layer to obtain a second prediction quality of the second image frame; A prediction weighting unit, configured to obtain a first evaluation weight of the first image frame and a second evaluation weight of the second image frame, weight the first prediction quality based on the first evaluation weight to obtain a first weighted prediction value, and weight the second prediction quality based on the second evaluation weight to obtain a second weighted prediction value; The quality determination unit is used to determine the sum of the first weighted prediction value and the second weighted prediction value as the predicted media quality of the multimedia data.

13. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program to execute the method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the method according to any one of claims 1 to 11 is executed.

Images