Evaluation method, model training method, device, medium and electronic device

Abstract

The embodiments of the present application provide an evaluation method, a model training method, an apparatus, a medium and an electronic device. Among them, the image quality evaluation method includes: obtaining an image to be processed whose quality needs to be evaluated; extracting the image features of the image to be processed through a feature extraction network, the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, the spatial attention module is used to extract the local features of the image to be processed in space; based on the image features, the image features are mapped through a fully connected layer to obtain a quality score of the image to be processed. The technical solution of the embodiments of the present application can improve the accuracy of image quality evaluation.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to an evaluation method, a model training method, a device, a medium and an electronic device. Background Art

[0002] In the process of image acquisition, processing, transmission and recording, due to the imperfections of imaging systems, processing methods, transmission media and recording equipment, as well as object movement, noise pollution and other reasons, some image distortion and degradation are inevitable, which makes it very difficult for people to understand the objective world and study and solve problems. With the development of artificial intelligence, related technologies have proposed to evaluate image quality based on artificial intelligence. However, the image quality assessment algorithm based on no reference has no reference image, making it difficult to quantify image distortion and degradation, which brings great challenges to quality assessment. Summary of the invention

[0003] The embodiments of the present application provide an evaluation method, a model training method, an apparatus, a medium and an electronic device, which can improve the accuracy of image quality evaluation at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a method for evaluating image quality is provided, comprising: obtaining an image to be processed whose quality needs to be evaluated; extracting image features of the image to be processed through a feature extraction network, the feature extraction network comprising a plurality of feature extraction units connected in sequence, each of the feature extraction units comprising a convolution block and a spatial attention module connected in sequence, the spatial attention module being used to extract local features of the image to be processed in space; based on the image features, mapping the image features through a fully connected layer to obtain a quality score of the image to be processed.

[0006] According to one aspect of an embodiment of the present application, a training method for an image quality assessment model includes: obtaining a training sample including a score value, wherein the score value is used to characterize the image quality of the training sample; extracting image features of the training sample through a feature extraction network included in the image quality assessment model, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, and the spatial attention module is used to extract local features of the training sample in space; based on the image features of the training sample, mapping the image features of the training sample through a fully connected layer included in the image quality assessment model to obtain a predicted score output by the image quality assessment model for the training sample; and adjusting parameters of the image quality assessment model according to the difference between the predicted score and the score value of the training sample to train the image quality assessment model.

[0007] According to one aspect of an embodiment of the present application, an image quality assessment device is provided, comprising: a first acquisition unit, configured to acquire an image to be processed whose quality needs to be assessed; a first processing unit, configured to extract image features of the image to be processed through a feature extraction network, the feature extraction network comprising a plurality of feature extraction units connected in sequence, each of the feature extraction units comprising a convolution block and a spatial attention module connected in sequence, the spatial attention module being used to extract local features of the image to be processed in space; a second processing unit, configured to map the image features through a fully connected layer based on the image features, to obtain a quality score of the image to be processed.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the spatial attention module includes a trunk branch for feature processing and a mask branch for extracting mask features; the processing process of the feature extraction unit includes: processing the output features of the convolution block in the feature extraction unit through a first number of residual units and then inputting them into the mask branch and the trunk branch respectively; fusing the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result; processing the fusion result through a second number of residual units to obtain the output features of the spatial attention module.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the mask branch includes a downsampling unit, a third number of residual units, an upsampling unit, a convolution unit and an activation function connected in sequence; wherein the upsampling unit is used to restore the size of the feature map processed by the third number of residual units to the size of the feature map input to the downsampling unit.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the downsampling unit includes: a first downsampling module, a fourth number of residual units, and a second downsampling module connected in sequence; the upsampling unit includes: a first upsampling module, a fifth number of residual units, and a second upsampling module connected in sequence.

[0011] In some embodiments of the present application, based on the aforementioned solution, the trunk branch includes a sixth number of residual units connected in sequence.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the feature extraction unit fuses the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result, including: calculating the element-wise product between the mask features and the output features of the trunk branch; calculating the sum of the element-wise product and the output features of the trunk branch, and using the sum as the fusion result between the mask features and the output features of the trunk branch.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the quality score of the image to be processed is evaluated by an image quality assessment model, and the image quality assessment model includes the feature extraction network and the fully connected layer, and the fully connected layer is connected to the feature extraction network.

[0014] According to one aspect of an embodiment of the present application, a training device for an image quality assessment model is provided, comprising: a second acquisition unit, configured to acquire a training sample including a score value, wherein the score value is used to characterize the image quality of the training sample; a third processing unit, configured to extract image features of the training sample through a feature extraction network included in the image quality assessment model, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, and the spatial attention module is used to extract local features of the training sample in space; a fourth processing unit, configured to map the image features of the training sample through a fully connected layer included in the image quality assessment model based on the image features of the training sample to obtain a predicted score output by the image quality assessment model for the training sample; and a fifth processing unit, configured to adjust parameters of the image quality assessment model according to the difference between the predicted score and the score value of the training sample to train the image quality assessment model.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit is configured to: extract the contour image of each preset target image; divide the each target image into multiple sub-images, and determine the corresponding area of ​​each sub-image in the contour image; according to the pixel values ​​of the pixel points contained in the corresponding area of ​​each sub-image in the contour image, select at least one sub-image from the multiple sub-images as a training sample, and the score value of the at least one sub-image is the same as the score value of the target image corresponding to the at least one sub-image.

[0016] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit is configured to: count the number of target pixel points in the corresponding area of ​​each sub-image in the contour image whose pixel values ​​are greater than a first set pixel value according to the pixel values ​​of the pixel points contained in the corresponding area of ​​each sub-image in the contour image; and select a sub-image whose number of target pixel points is greater than or equal to the set number from the multiple sub-images as the training sample.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit is also configured to: before selecting at least one sub-image from the multiple sub-images as a training sample, set the pixel value of the pixel point in the contour image whose pixel value is less than or equal to a second set pixel value to a first value, and the second set pixel value is less than or equal to the first set pixel value.

[0018] In some embodiments of the present application, based on the aforementioned solution, the second acquisition unit is further configured to: perform Gaussian blur processing on each target image before extracting the contour image of each target image.

[0019] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit is further configured to: acquire a target video containing a scoring value; capture an image frame in the target video as the target image, and the scoring value of the target image is the same as the scoring value of the target video.

[0020] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image quality assessment method as described in the above embodiments is implemented, or the image quality assessment model training method as described in the above embodiments is implemented.

[0021] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image quality assessment method as described in the above embodiments, or implement the image quality assessment model training method as described in the above embodiments.

[0022] According to one aspect of an embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image quality assessment method provided in the above-mentioned various optional embodiments, or implements the training method of the image quality assessment model as described in the above-mentioned embodiments.

[0023] In the technical solutions provided in some embodiments of the present application, image features of an image to be processed are extracted by a feature extraction network, and the feature extraction network includes a plurality of feature extraction units connected in sequence, each feature extraction unit includes a convolution block and a spatial attention module connected in sequence, and the spatial attention module is used to extract the local features of the image to be processed in space, and then the extracted image features are mapped through a fully connected layer to obtain a quality score of the image to be processed, so that the spatial attention module can be used to increase the attention to distorted areas and degraded areas of the image, thereby improving the accuracy of image quality assessment.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0027] Figure 2 A schematic diagram showing the placement of a video encoding device and a video decoding device in a streaming transmission system;

[0028] Figure 3A schematic diagram of a scheme for evaluating image quality is shown;

[0029] Figure 4 A flowchart of a method for evaluating image quality according to an embodiment of the present application is shown;

[0030] Figure 5 A structural diagram of a feature extraction network according to an embodiment of the present application is shown;

[0031] Figure 6 A structural diagram of a spatial attention module according to an embodiment of the present application is shown;

[0032] Figure 7 A flowchart of training an image quality assessment model according to an embodiment of the present application is shown;

[0033] Figure 8 A structural diagram of an image quality assessment model according to an embodiment of the present application is shown;

[0034] Fig. 9 A comparison chart of the scoring values ​​after the image quality is evaluated using the technical solution of the embodiment of the present application is shown;

[0035] Fig.10 A block diagram of an image quality evaluation device according to an embodiment of the present application is shown;

[0036] Fig.11 A block diagram of a training device for an image quality assessment model according to an embodiment of the present application is shown;

[0037] Fig.12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0039] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0040] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0043] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0044] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0045] Computer vision technology (CV for short) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0046] The solution provided in the embodiments of the present application involves technologies such as artificial intelligence computer vision, which are specifically described by the following embodiments:

[0047] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0048] like Figure 1 As shown, the system architecture 100 includes a plurality of terminal devices, which can communicate with each other through, for example, a network 150. For example, the system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected through the network 150. Figure 1 In the embodiment of the present invention, the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0049] For example, the first terminal device 110 may encode video data (e.g., a video picture stream captured by the terminal device 110) for transmission to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device 120 may receive the encoded video data from the network 150, decode the encoded video data to restore the video data, and display the video picture based on the restored video data.

[0050] In one embodiment of the present application, the system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, and the bidirectional transmission may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device 130 and the fourth terminal device 140 through the network 150. Each of the third terminal device 130 and the fourth terminal device 140 may also receive the encoded video data transmitted by the other terminal device of the third terminal device 130 and the fourth terminal device 140, and may decode the encoded video data to restore the video data, and may display the video picture on an accessible display device according to the restored video data, and the quality of the restored video picture affects the effect of displaying on the display device, so the quality of the video picture needs to be evaluated in the embodiment of the present application.

[0051] exist Figure 1 In the embodiment of the present invention, the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140 may be servers, smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, etc., but the principles disclosed in the present application are not limited thereto. The server in the embodiment of the present application may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0052] The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140, including, for example, wired and / or wireless communication networks. The communication network 150 can exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network 150 may be irrelevant to the operation disclosed in this application.

[0053] In one embodiment of the present application, Figure 2The placement of the video encoding device and the video decoding device in the streaming environment is shown. The subject matter disclosed in the present application can be equally applicable to other video-supported applications, including, for example, video conferencing, digital TV (television), storing compressed video on digital media including CD, DVD, memory stick, etc.

[0054] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, which creates an uncompressed video picture stream 202. In an embodiment, the video picture stream 202 includes samples taken by a digital camera. The video picture stream 202 is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video data 204 (or the encoded video code stream 204), and the video picture stream 202 can be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream 202, the encoded video data 204 (or the encoded video code stream 204) is depicted as a thin line to emphasize the lower data volume of the encoded video data 204 (or the encoded video code stream 204), which can be stored on the streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2 The client subsystem 206 and the client subsystem 208 in the streaming server 205 can access the streaming server 205 to retrieve the copy 207 and the copy 209 of the encoded video data 204. The client subsystem 206 may include, for example, a video decoding device 210 in the electronic device 230. The video decoding device 210 decodes the incoming copy 207 of the encoded video data and generates an output video picture stream 211 that can be presented on a display 212 (e.g., a display screen) or another presentation device. In some streaming systems, the encoded video data 204, the video data 207, and the video data 209 (e.g., a video bitstream) may be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application may be used in the context of the VVC standard.

[0055] It should be noted that the electronic device 220 and the electronic device 230 may include other components not shown in the figure. For example, the electronic device 220 may include a video decoding device, and the electronic device 230 may also include a video encoding device.

[0056] In order to evaluate the quality of video images, in a technical solution, such as Figure 3 As shown, the image is first distorted to different degrees by Gaussian blur to obtain images with known relative image quality, and then a Siamese neural network is trained to sort these images according to quality. This method can automatically generate a sorted image set, thereby avoiding the heavy manual labeling work. Then, the information in the trained Siamese neural network is transferred to a normal network by fine-tuning, and the normal network can be a CNN (Convolutional Neural Networks) network. Then, the image quality of the distorted image with a smaller data set (the distorted image already contains the quality score) is evaluated by the normal network, so as to achieve the evaluation of image quality by training and fine-tuning the normal network.

[0057] However, image quality assessment is a very broad problem, because there are many reasons that cause image distortion, and it is difficult for algorithms to cover all of them. Figure 3 The technical solution shown is only for training and processing a certain type of noise, and is prone to poor generalization for actual images. At the same time, due to the high computational complexity of the twin neural network, the training process is relatively complicated.

[0058] Based on this, the present application embodiment proposes a new image quality evaluation method, specifically as follows Figure 4 As shown, the image quality evaluation method can be executed by a device with a computing and processing function, such as a server. Figure 4 As shown, the image quality evaluation method at least includes steps S410 to S430, which are described in detail as follows:

[0059] In step S410, an image to be processed whose quality needs to be evaluated is obtained.

[0060] In one embodiment of the present application, the image to be processed may be obtained through a network, such as based on Figure 1 or Figure 2 The image to be processed may also be an image acquired by an image acquisition device, such as an image acquired by a camera, a video surveillance device, etc. The image to be processed may also be a computer-generated or synthesized image.

[0061] In step S420, image features of the image to be processed are extracted through a feature extraction network, where the feature extraction network includes a plurality of feature extraction units connected in sequence, each feature extraction unit includes a convolution block and a spatial attention module connected in sequence, and the spatial attention module is used to extract local features of the image to be processed in space.

[0062] In one embodiment of the present application, Figure 5 As shown, the feature extraction network includes a plurality of feature extraction units 501 connected in sequence, that is, the output of the previous feature extraction unit is used as the input of the next feature extraction unit, and each feature extraction unit includes a convolution block 5011 and a spatial attention module 5012 connected in sequence. The convolution block 5011 is used to perform convolution processing on the features, and the spatial attention module 5012 is used to extract the local features of the image to be processed in space, thereby increasing the attention to the distorted and degraded areas of the image, which is beneficial to improving the accuracy of image quality assessment.

[0063] In one embodiment of the present application, Figure 6 As shown, the spatial attention module includes a trunk branch 602 for feature processing and a mask branch 603 for extracting mask features. The processing process of the spatial attention module mainly includes: taking the output features of the convolution block in the feature extraction unit as the input features x of the spatial attention module, and the input features x are passed through a first number of residual units (i.e. Figure 6 601) are processed and then input to the mask branch 603 and the trunk branch 602 respectively, and then the mask features output by the mask branch 603 and the output features of the trunk branch 602 are fused to obtain a fusion result, and then the fusion result is passed through a second number of residual units (i.e. Figure 6 604) shown in , and the output feature y of the spatial attention module is obtained.

[0064] In one embodiment of the present application, the mask branch 603 includes a down-sampling unit 6031, a third number of residual units (i.e. Figure 6 6032), an up-sampling unit 6033, a convolution unit 6034 and an activation function 6035. The up-sampling unit 6033 is used to restore the size of the feature map processed by the third number of residual units to the size of the feature map input to the down-sampling unit 6031.

[0065] In one embodiment of the present application, Figure 6 As shown, the down-sampling unit 6031 includes: a first down-sampling module, a fourth number of residual units and a second down-sampling module connected in sequence. The up-sampling unit 6033 includes: a first up-sampling module, a fifth number of residual units and a second up-sampling module connected in sequence.

[0066] In one embodiment of the present application, Figure 6 As shown, the trunk branch 602 includes a sixth number of residual units connected in sequence.

[0067] In one embodiment of the present application, Figure 6As shown, when the mask feature M(x) output by the mask branch 603 and the output feature T(x) of the trunk branch 602 are fused, the specific fusion process can be: calculating the element product between the mask feature M(x) and the output feature T(x) of the trunk branch 602, and then calculating the sum of the element product and the output feature T(x) of the trunk branch 602, and using the sum as the fusion result H(x) between the mask feature and the output feature of the trunk branch.

[0068] Continue to refer to Figure 4 As shown, in step S430, based on the image features of the image to be processed, the image features are mapped through a fully connected layer to obtain a quality score of the image to be processed.

[0069] In one embodiment of the present application, Figure 5 As shown, the output features of the feature extraction network are input to the fully connected layer 502, and then the quality score of the image to be processed is obtained through the mapping processing of the fully connected layer 502.

[0070] In one embodiment of the present application, the image to be processed can be evaluated for quality by an image quality assessment model including the above-mentioned feature extraction network and a fully connected layer, wherein the input of the fully connected layer is connected to the output of the feature extraction network. In this case, the image quality assessment model can also be trained by a target image including a scoring value.

[0071] In one embodiment of the present application, a target video including a scoring value may be obtained, and then an image frame in the target video may be captured as a target image, and the scoring value of the target image may be the same as the scoring value of the target video.

[0072] In one embodiment of the present application, Figure 7 As shown, the process of training the image quality assessment model using the target image containing the scoring value may include the following steps S710 to S740, which are specifically described as follows:

[0073] In step S710 , a contour image of each target image is extracted.

[0074] In one embodiment of the present application, the Sobel operator may be used to perform edge extraction to obtain the contour image of the target image. Before extracting the contour image of each target image, Gaussian blur processing may be performed on each target image.

[0075] In step S720, each target image is divided into a plurality of sub-images, and a corresponding area of ​​each sub-image in the contour image is determined.

[0076] In one embodiment of the present application, the target image can be divided into a plurality of sub-images according to a set size, and optionally, the sub-images may not overlap each other. The set size can be selected according to the input requirements of the model, for example, the set size can be 224×224. Since there is a corresponding relationship between the contour image and the target image, after the target image is divided into a plurality of sub-images, the corresponding area of ​​each sub-image in the contour image can be determined according to the corresponding relationship.

[0077] In step S730, at least one sub-image is selected from multiple sub-images as a training sample based on the pixel values ​​of the pixel points contained in the corresponding areas of each sub-image in the contour image, and the score value of the at least one sub-image is the same as the score value of the target image corresponding to the at least one sub-image.

[0078] In one embodiment of the present application, the number of target pixels whose pixel values ​​are greater than a first set pixel value in the corresponding area of ​​each sub-image in the contour image can be counted according to the pixel values ​​of the pixels contained in the corresponding area of ​​each sub-image in the contour image, and then a sub-image whose number of target pixels is greater than or equal to the set number is selected from multiple sub-images as a training sample. This method can ensure that the selected training samples have more effective features, thereby improving the training effect of the image quality assessment model.

[0079] In one embodiment of the present application, before selecting at least one sub-image from a plurality of sub-images as a training sample, the pixel value of a pixel point in the contour image whose pixel value is less than or equal to a second set pixel value may be set to a first value, and the second set pixel value is less than or equal to the first set pixel value. The technical solution of this embodiment makes it easy to count the number of target pixel points whose pixel values ​​are greater than the first set pixel value. Optionally, the second set pixel value may be 0, and the first set pixel value may be a non-zero value.

[0080] In step S740, the image quality assessment model is trained using training samples.

[0081] In one embodiment of the present application, when training an image quality assessment model, the image features of the training samples can be extracted through a feature extraction network included in the image quality assessment model, wherein the structure of the feature extraction network is as described in the above embodiment, wherein the spatial attention module is used to extract the local features of the training samples in space, and then based on the image features of the training samples, the image features of the training samples are mapped through a fully connected layer included in the image quality assessment model to obtain a predicted score output by the image quality assessment model for the training samples, and finally, based on the difference between the predicted score and the score value of the training samples, the parameters of the image quality assessment model are adjusted to achieve training of the image quality assessment model.

[0082] In one embodiment of the present application, when training the image quality assessment model, the loss function may use MSE (Mean Squared Error). Specifically, Among them, ω i Represents a constant greater than 0, y i represents the predicted score output by the image quality assessment model for the i-th training sample, Indicates the score value of the i-th training sample. Of course, in other embodiments of the present application, other loss functions may also be selected, such as SSE (Sumof Squares for Error, and variance), RMSE (Root Mean Squared Error, standard deviation), etc. The image quality assessment model may be optimized using the Adam optimizer during the training process.

[0083] In passing Figure 7 After the technical solution of the embodiment shown trains the image quality assessment model, the image quality assessment model can be used to implement image quality assessment processing. For example, the image to be assessed can be input into the image quality assessment model, and then the score value corresponding to the image to be assessed by the image quality assessment model can be obtained.

[0084] Optionally, the technical solution of the embodiment of the present application can be applied to occasions including image compression, video encoding and decoding, video monitoring, etc., or as a module for evaluating the quality of images (such as the degree of image distortion) in some specific applications. For example, when systems such as remote conferencing and video on demand are affected by adverse factors such as transmission errors and network delays, it is necessary to monitor the image quality online in real time so that the service provider can dynamically adjust the source positioning strategy to meet the service quality requirements. In the video processing of the video cloud, it is necessary to perform quality assessment on massive videos to facilitate classification processing. Among them, the evaluation results of the image can be characterized by the MOS (mean opinion score, average subjective score) value, and the larger the value, the better the image quality.

[0085] The following combination Figure 8 and Figure 6 , the specific processing process of an embodiment of the present application is described in detail:

[0086] like Figure 8 As shown, in one embodiment of the present application, image quality assessment can be performed on an input image in any format (such as png format, bmp format, jpg format, etc.), and the input image can be an image of RGB channels. Optionally, the input image can be preprocessed first, such as Gaussian blur processing, and then input into the convolution block to extract features. Figure 8 As shown, a spatial attention module is added after convolution block 1, and the output of the spatial attention module is used as the input of convolution block 2. In this way, the combination of convolution blocks and spatial attention modules is continuously stacked. In this embodiment, a combination of 4 convolution blocks and 4 spatial attention modules is stacked as an example, and then a fully connected layer is used after convolution block 5 to output the MOS value of the input image. Of course, in other embodiments of the present application, a spatial attention module can be added after convolution block 5, and then a fully connected layer is used to output the MOS value of the input image.

[0087] In one embodiment of the present application, the image quality can be evaluated based on the MobileNet network structure. The following is a detailed description based on the MobileNetV1 network structure as an example, wherein the structure of the image quality evaluation model is as follows: Figure 8 As shown in Table 1, when the size of the feature map output by the convolution block changes from 112×112×64 to 56×56×64, a spatial attention module is added. The spatial attention module requires additional convolutional layers for assistance. The specific structure can be referred to Figure 6 As shown. Figure 8 In the structure shown, the convolution block extracts the features of the image from shallow to deep, while the spatial attention module strengthens the areas with obvious compression distortion in the continuous optimization process, further improving the prediction ability of the network. Then, the first fully connected layer of MobileNetV1 is used for dimensionality reduction, reducing the dimension from 1024 to 128, and then from 128 to 1, and the output is the MOS value of the image. Among them, the loss function used in the image quality assessment model during training can be MSE, and the optimizer can be Adam. Alternatively, the optimizer can also be replaced by other methods such as SGD (Stochastic Gradient Descent).

[0088]

[0089]

[0090] Table 1

[0091] In one embodiment of the present application, Figure 6 As shown in FIG. 6 , the spatial attention module is divided into two branches: a mask branch 603 and a trunk branch 602. After the input feature x passes through the spatial attention module, its feature size and number of channels remain unchanged. The trunk branch 602 is used for feature processing and can be improved into any state-of-the-art network structure. After the input feature x passes through the trunk branch 602, the output is T(x). The trunk branch 602 can be composed of t residual units, that is, Figure 6 The 602 shown in FIG. 6 includes t residual units.

[0092] The mask branch 603 can use a coding-decoding structure to learn a soft weighted mask M(x) of the same size as T(x). The coding-decoding structure simulates a fast feedforward and feedback attention process. The output mask feature M(x) is used to control the neuron gate of the trunk branch 602. The output of the mask branch is M(x). In the spatial attention module, the output of the trunk branch 602 and the mask branch 603 are fused to obtain H(x) = (1 + M(x)) × T(x), and then passed through p residual units (i.e. Figure 6 604 shown in FIG. 604 includes p residual units) to obtain the final output feature y.

[0093] In one embodiment of the present application, the number of the fourth number of residual units included in the down-sampling unit 6031 in the mask branch 603 may be r, Figure 6 The number of the third number of residual units included in 6032 shown in the figure may be 2r, and the number of the fifth number of residual units included in the up-sampling unit 6033 may also be r.

[0094] Optionally, in one embodiment of the present application, the settings of the three hyperparameters can be {p=1, t=2, r=1}, or adjusted according to experimental results. The design of the residual unit can adopt a combination of convolutional layers, such as {convolutional layer with a convolution kernel size of 1×1, convolutional layer with a convolution kernel size of 3×3, and convolutional layer with a convolution kernel size of 1×1}, wherein the number of channels of the convolutional layer with a convolution kernel size of 1×1 is consistent with the number of channels c of the input feature x, and the number of channels of the convolutional layer with a convolution kernel size of 3×3 can be 2c, that is, first increase the dimension and then reduce the dimension. Of course, convolution kernel sizes of other sizes can also be used, but the number of channels of the output feature y needs to be kept consistent with the number of channels of the input feature x.

[0095] In the mask branch 603, the input feature x is first downsampled by the max pooling method, and the stride can be 2, and then max pooling is performed again after r residual units, and the size of the overall feature map is reduced by 4 times, and then upsampled by 2 times after 2r residual units, and upsampled by 2 times again after r residual units to restore the original map size. The purpose of downsampling is to obtain more global semantic information, and the upsampling recovery size is to generate a mask of the same size as the input feature x. Figure 6 The 6034 shown in the figure can contain two convolutional layers. Both of these convolutional layers can make the convolution kernel size 1×1. Its function is to improve the spatial expression of the spatial attention module. The number of channels is adjusted according to the actual situation. For example, the number of channels of the first convolutional layer in 6034 can be 2c, and the number of channels of the second convolutional layer can be c. The activation function 6035 can use the sigmoid activation function, and then output the normalized M(x). Based on Figure 6 In the spatial attention module shown, when M(x) is equal to 0, the mask branch 603 has no effect on T(x), and when M(x) is greater than 0, T(x) is strengthened, thereby enhancing the expressiveness of the feature in space.

[0096] When training the image quality assessment model, it is necessary to select appropriate training samples. In one embodiment of the present application, a video that has been subjectively scored can be obtained, and then the video frame can be extracted as a training sample. For each frame of image I, Gaussian blur processing is first performed on image I, and then the Sobel operator is used to extract the edge to obtain the contour image M, and a threshold m is set, and each pixel point of the contour image M is judged separately, and the value of the pixel point whose pixel value is lower than m is set to 0. Since the input image size requirement of mobilenetV1 is 224×224, the image I can be divided into non-overlapping sub-blocks, and the size of each sub-block is 224×224. Then, according to the corresponding position of each sub-block of image I on the contour image M, the number of non-zero pixels in each sub-block on the contour image M is determined, and then the sub-blocks with a number of non-pixel points greater than n (n is a set threshold, for example, it can be set to n=5000) are selected as training sets, and the mos values ​​of all selected sub-blocks of the same video are the same as the mos values ​​of the video. Then, the image quality assessment model can be trained by the selected training set.

[0097] The specific values ​​in the above embodiments of the present application are only examples and can be adjusted according to actual conditions during implementation. For example, the hyperparameters in the spatial attention module can be set according to actual conditions, or the structure of the residual unit can be adjusted, and the trunk branch 602 can also be replaced by other network models, such as vggnet, Alexnet, etc.

[0098] Through the technical solution of the above embodiment of the present application, a relatively reasonable score can be predicted for images of different qualities. For an image with greater compression noise, the score is lower. For example, for Fig. 9 In the two images shown in , the image on the left has higher quality, so the score is higher; while the image on the right has lower quality, so the score is lower. In the test set, there are 848 test videos, and one frame is taken every 10 frames (each video is about 120-150 frames). The experimental results are as follows: PLCC (Pearson linear correlation coefficient) is 0.864, and SROCC (Spearman rank order correlation coefficient) is 0.867. Judging from these two indicators, the technical solution of the embodiment of this application has better effect on non-reference image quality assessment on this data set.

[0099] The following describes an apparatus embodiment of the present application, which can be used to perform the image quality evaluation method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the embodiment of the image quality evaluation method in the above-mentioned embodiment of the present application.

[0100] Fig.10 A block diagram of an image quality assessment device according to an embodiment of the present application is shown. The image quality assessment device may be arranged in a device having a computing and processing function, such as a server.

[0101] Reference Fig.10 As shown, according to an embodiment of the present application, an image quality assessment device 1000 includes: an acquisition unit 1002 , a first processing unit 1004 , and a second processing unit 1006 .

[0102] Among them, the acquisition unit 1002 is configured to acquire an image to be processed whose quality needs to be evaluated; the first processing unit 1004 is configured to extract image features of the image to be processed through a feature extraction network, and the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, and the spatial attention module is used to extract local features of the image to be processed in space; the second processing unit 1006 is configured to map the image features through a fully connected layer based on the image features to obtain a quality score of the image to be processed.

[0103] In some embodiments of the present application, based on the aforementioned scheme, the spatial attention module includes a trunk branch for feature processing and a mask branch for extracting mask features; the processing process of the feature extraction unit includes: processing the output features of the convolution block in the feature extraction unit through a first number of residual units and then inputting them into the mask branch and the trunk branch respectively; fusing the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result; processing the fusion result through a second number of residual units to obtain the output features of the spatial attention module.

[0104] In some embodiments of the present application, based on the aforementioned scheme, the mask branch includes a downsampling unit, a third number of residual units, an upsampling unit, a convolution unit and an activation function connected in sequence; wherein the upsampling unit is used to restore the size of the feature map processed by the third number of residual units to the size of the feature map input to the downsampling unit.

[0105] In some embodiments of the present application, based on the aforementioned scheme, the downsampling unit includes: a first downsampling module, a fourth number of residual units, and a second downsampling module connected in sequence; the upsampling unit includes: a first upsampling module, a fifth number of residual units, and a second upsampling module connected in sequence.

[0106] In some embodiments of the present application, based on the aforementioned solution, the trunk branch includes a sixth number of residual units connected in sequence.

[0107] In some embodiments of the present application, based on the aforementioned scheme, the feature extraction unit fuses the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result, including: calculating the element-wise product between the mask features and the output features of the trunk branch; calculating the sum of the element-wise product and the output features of the trunk branch, and using the sum as the fusion result between the mask features and the output features of the trunk branch.

[0108] In some embodiments of the present application, based on the aforementioned scheme, the quality score of the image to be processed is evaluated by an image quality assessment model, and the image quality assessment model includes the feature extraction network and the fully connected layer, and the fully connected layer is connected to the feature extraction network.

[0109] Fig.11 A block diagram of a training device for an image quality assessment model according to an embodiment of the present application is shown. The training device for the image quality assessment model can be set in a device with a computing and processing function, such as a server.

[0110] Reference Fig.11 As shown, a training device 1100 for an image quality assessment model according to an embodiment of the present application includes: a second acquisition unit 1102 , a third processing unit 1104 , a fourth processing unit 1106 and a fifth processing unit 1108 .

[0111] Among them, the second acquisition unit 1102 is configured to acquire a training sample including a score value, wherein the score value is used to characterize the image quality of the training sample; the third processing unit 1104 is configured to extract the image features of the training sample through a feature extraction network included in an image quality assessment model, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, and the spatial attention module is used to extract the local features of the training sample in space; the fourth processing unit 1106 is configured to map the image features of the training sample through a fully connected layer included in the image quality assessment model based on the image features of the training sample to obtain a predicted score output by the image quality assessment model for the training sample; the fifth processing unit 1108 is configured to adjust the parameters of the image quality assessment model according to the difference between the predicted score and the score value of the training sample to train the image quality assessment model.

[0112] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit 1102 is configured to: extract the contour image of each preset target image; divide the each target image into multiple sub-images, and determine the corresponding area of ​​each sub-image in the contour image; select at least one sub-image from the multiple sub-images as a training sample according to the pixel values ​​of the pixel points contained in the corresponding area of ​​each sub-image in the contour image, and the score value of the at least one sub-image is the same as the score value of the target image corresponding to the at least one sub-image.

[0113] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit 1102 is configured to: count the number of target pixel points in the corresponding area of ​​each sub-image in the contour image whose pixel values ​​are greater than a first set pixel value according to the pixel values ​​of the pixel points contained in the corresponding area of ​​each sub-image in the contour image; and select a sub-image whose number of target pixel points is greater than or equal to the set number from the multiple sub-images as the training sample.

[0114] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit 1102 is also configured to: before selecting at least one sub-image from the multiple sub-images as a training sample, set the pixel value of the pixel point in the contour image whose pixel value is less than or equal to the second set pixel value to a first value, and the second set pixel value is less than or equal to the first set pixel value.

[0115] In some embodiments of the present application, based on the aforementioned solution, the second acquisition unit 1102 is further configured to: perform Gaussian blur processing on each target image before extracting the contour image of each target image.

[0116] In some embodiments of the present application, based on the aforementioned scheme, the second acquisition unit 1102 is further configured to: acquire a target video containing a scoring value; capture an image frame in the target video as the target image, and the scoring value of the target image is the same as the scoring value of the target video.

[0117] Fig.12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0118] It should be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method described in the above embodiment. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202 and the RAM 1203 are connected to each other through the bus 1204. The input / output (I / O) interface 1205 is also connected to the bus 1204.

[0120] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0121] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0122] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0123] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0125] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0126] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0127] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0128] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0129] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for evaluating image quality, characterized in that: include: Obtaining an image to be processed whose quality needs to be evaluated; Extracting image features of the image to be processed through a feature extraction network, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, the spatial attention module is used to extract local features of the image to be processed in space, and the spatial attention module includes a trunk branch for feature processing and a mask branch for extracting mask features; Based on the image features, mapping the image features through a fully connected layer to obtain a quality score of the image to be processed; The processing of the feature extraction unit includes: After processing the output features of the convolution block in the feature extraction unit through a first number of residual units, the output features are input to the mask branch and the trunk branch respectively; Fusing the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result; The fusion result is processed by a second number of residual units to obtain output features of the spatial attention module.

2. The image quality evaluation method according to claim 1, characterized in that: The mask branch includes a downsampling unit, a third number of residual units, an upsampling unit, a convolution unit and an activation function connected in sequence; The up-sampling unit is used to restore the size of the feature map processed by the third number of residual units to the size of the feature map input to the down-sampling unit.

3. The image quality evaluation method according to claim 2, characterized in that: The downsampling unit comprises: a first downsampling module, a fourth number of residual units and a second downsampling module connected in sequence; The up-sampling unit includes: a first up-sampling module, a fifth number of residual units and a second up-sampling module connected in sequence.

4. The image quality evaluation method according to claim 1, characterized in that: The trunk branch includes a sixth number of residual units connected in sequence.

5. The image quality evaluation method according to claim 1, characterized in that: The mask features output by the mask branch and the output features of the trunk branch are fused to obtain a fusion result, including: Calculating the element-wise product between the mask feature and the output feature of the trunk branch; A sum value between the element product and the output feature of the trunk branch is calculated, and the sum value is used as a fusion result between the mask feature and the output feature of the trunk branch.

6. The image quality evaluation method according to any one of claims 1 to 5, characterized in that: The quality score of the image to be processed is evaluated by an image quality assessment model, wherein the image quality assessment model includes the feature extraction network and the fully connected layer, and the fully connected layer is connected to the feature extraction network.

7. A training method for an image quality assessment model, characterized in that: include: Acquire a training sample including a scoring value, wherein the scoring value is used to characterize the image quality of the training sample; Extracting image features of the training sample through a feature extraction network included in the image quality assessment model, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, the spatial attention module is used to extract local features of the training sample in space, and the spatial attention module includes a trunk branch for feature processing and a mask branch for extracting mask features; Based on the image features of the training samples, mapping processing is performed on the image features of the training samples through a fully connected layer included in the image quality assessment model to obtain a prediction score output by the image quality assessment model for the training samples; According to the difference between the predicted score and the score value of the training sample, adjusting the parameters of the image quality assessment model to train the image quality assessment model; The processing of the feature extraction unit includes: After processing the output features of the convolution block in the feature extraction unit through a first number of residual units, the output features are input to the mask branch and the trunk branch respectively; Fusing the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result; The fusion result is processed by a second number of residual units to obtain output features of the spatial attention module.

8. The method for training an image quality assessment model according to claim 7, characterized in that: Get training samples with scoring values, including: Extracting contour images of each preset target image; Dividing each target image into a plurality of sub-images, and determining a corresponding area of ​​each sub-image in the contour image; According to the pixel values ​​of the pixel points contained in the corresponding areas of each sub-image in the contour image, at least one sub-image is selected from the multiple sub-images as a training sample, and the score value of the at least one sub-image is the same as the score value of the target image corresponding to the at least one sub-image.

9. The method for training an image quality assessment model according to claim 8, characterized in that: Selecting at least one sub-image from the plurality of sub-images as a training sample according to pixel values ​​of pixels contained in corresponding areas of the respective sub-images in the contour image comprises: According to the pixel values ​​of the pixels contained in the corresponding area of ​​each sub-image in the contour image, counting the number of target pixel points whose pixel values ​​in the corresponding area of ​​each sub-image in the contour image are greater than a first set pixel value; A sub-image having the number of target pixels greater than or equal to a set number is selected from the multiple sub-images as the training sample.

10. The method for training an image quality assessment model according to claim 9, characterized in that: Before selecting at least one sub-image from the plurality of sub-images as a training sample, the training method further includes: The pixel values ​​of the pixels in the contour image whose pixel values ​​are less than or equal to the second set pixel value are set to the first value, and the second set pixel value is less than or equal to the first set pixel value.

11. The method for training an image quality assessment model according to claim 8, characterized in that: Also includes: Get the target video containing the scoring value; An image frame in the target video is intercepted as the target image, and the score value of the target image is the same as the score value of the target video.

12. An image quality assessment device, characterized in that: include: A first acquisition unit, configured to acquire an image to be processed whose quality needs to be evaluated; A first processing unit is configured to extract image features of the image to be processed through a feature extraction network, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, the spatial attention module is used to extract local features of the image to be processed in space, and the spatial attention module includes a trunk branch for feature processing and a mask branch for extracting mask features; A second processing unit is configured to perform mapping processing on the image features through a fully connected layer based on the image features to obtain a quality score of the image to be processed; The processing of the feature extraction unit includes: After processing the output features of the convolution block in the feature extraction unit through a first number of residual units, the output features are input to the mask branch and the trunk branch respectively; Fusing the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result; The fusion result is processed by a second number of residual units to obtain output features of the spatial attention module.

13. A training device for an image quality assessment model, characterized in that: include: A second acquisition unit is configured to acquire a training sample including a scoring value, where the scoring value is used to characterize the image quality of the training sample; a third processing unit, configured to extract image features of the training sample through a feature extraction network included in the image quality assessment model, wherein the feature extraction network includes a plurality of feature extraction units connected in sequence, each of the feature extraction units includes a convolution block and a spatial attention module connected in sequence, the spatial attention module is used to extract local features of the training sample in space, and the spatial attention module includes a trunk branch for feature processing and a mask branch for extracting mask features; a fourth processing unit configured to perform mapping processing on the image features of the training samples through a fully connected layer included in the image quality assessment model based on the image features of the training samples, so as to obtain a prediction score output by the image quality assessment model for the training samples; a fifth processing unit, configured to adjust parameters of the image quality assessment model according to a difference between the predicted score and the score value of the training sample, so as to train the image quality assessment model; The processing of the feature extraction unit includes: After processing the output features of the convolution block in the feature extraction unit through a first number of residual units, the output features are input to the mask branch and the trunk branch respectively; Fusing the mask features output by the mask branch and the output features of the trunk branch to obtain a fusion result; The fusion result is processed by a second number of residual units to obtain output features of the spatial attention module.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image quality assessment method according to any one of claims 1 to 6 is implemented, or the image quality assessment model training method according to any one of claims 7 to 11 is implemented.

15. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more computer programs, which, when executed by the one or more processors, enables the one or more processors to implement the image quality assessment method as described in any one of claims 1 to 6, or implement the image quality assessment model training method as described in any one of claims 7 to 11.

16. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. The processor of the electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the image quality assessment method described in any one of claims 1 to 6, or implements the image quality assessment model training method described in any one of claims 7 to 11.

Images