Image quality scoring method, apparatus, system, storage medium, and electronic device

Abstract

The application discloses an image quality scoring method, comprising: inputting an original image to be scored and a label corresponding to each current service type into a pre-trained state recognition model, marking an image area related to each current service type, and performing feature extraction based on the image area to obtain a first state feature vector corresponding to each current service type; for each first state feature vector, calculating the similarity of the first state feature vector with a state feature vector of R*P standard images corresponding to the corresponding current service type respectively to obtain R*P similarity results; and for each first state feature vector, determining a quality score result of the original image to be scored under the corresponding current service type based on the R*P similarity results. According to the application, different service types can be correspondingly matched to perform reasonable image quality judgment, and the quality judgment result is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology field, and particularly relates to an image quality scoring method, device, system, storage medium and electronic equipment for state recognition. BACKGROUND

[0002] With the progress of image processing technology, the technology of state recognition using images has been more and more widely applied.

[0003] When the image quality is poor, the performance of the state recognition system will be affected, resulting in state recognition errors. For example, the driving cabin illegal behavior judgment is a typical application of state recognition using images. When the driving cabin image collected is poor, the performance of the driving cabin illegal behavior judgment model will be affected, resulting in illegal behavior false positives, which requires a large amount of manpower for review and reduces the income. Therefore, it is hoped that the quality scoring process will be introduced before the state recognition process to filter out images with poor quality and improve the accuracy of state recognition. SUMMARY

[0004] The present application provides an image quality scoring method, device, system, storage medium and electronic equipment for state recognition, which can reasonably score the image quality for different business types and improve the accuracy of state recognition.

[0005] To achieve the above object, the present application adopts the following technical solutions:

[0006] The present application provides an image quality scoring method, which comprises:

[0007] The original image to be scored and the label corresponding to each current business type are input into a pre-trained state recognition model, the image area related to each current business type is marked, and feature extraction is performed based on the image area to obtain a first state feature vector corresponding to each current business type;

[0008] For each first state feature vector, the similarity of the first state feature vector to the state feature vectors of R*P standard images corresponding to the corresponding current business type is calculated, and R*P similarity results are obtained;

[0009] For each first state feature vector, based on the R*P similarity results, the quality scoring result of the original image to be scored under the corresponding current business type is determined;

[0010] The state feature vector of the standard image is a state feature vector obtained by inputting a standard image and a label of a corresponding current service type into the state recognition model for feature extraction.

[0011] Preferably, the state recognition model comprises a first hierarchical network for semantic segmentation and a second hierarchical network for state recognition.

[0012] The input of the first hierarchical network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to the current service type.

[0013] The second hierarchical network comprises M sub-networks corresponding to M service types; wherein M is the total number of all service types.

[0014] The input of each sub-network is the original image to be scored and the corrected semantic feature map corresponding to each label of the first current service type corresponding to the sub-network. Each sub-network connects the original image to be scored and the corrected semantic feature map in the channel dimension, and inputs the connected image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein the corrected semantic feature map is a semantic feature map up-sampled to the same size as the original image to be scored.

[0015] Preferably, the inputting the connected image into the classification network for state recognition comprises:

[0016] dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each two-dimensional slice in each two-dimensional slice;

[0017] inputting all two-dimensional slices with added two-dimensional position information into the classification network for state recognition;

[0018] wherein, through the training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current service type is ensured to be greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current service type.

[0019] Preferably, an attention mechanism is added in the classification network, and a weight is assigned to each pixel input into the classification network, and the weight of a pixel located in the image region related to the current business type is greater than the weight of a pixel located outside the image region related to the current business type.

[0020] Preferably, when training the state recognition model, the first hierarchical network is trained by using training sample images, and the second hierarchical network is trained based on the trained first hierarchical network; or,

[0021] When training the state recognition model, the first hierarchical network and the second hierarchical network are jointly trained; or,

[0022] When training the state recognition model, the first hierarchical network is initially trained by using training sample images, and the second hierarchical network is initially trained based on the first hierarchical network obtained after initial training; and after the initial training of the second hierarchical network, the first hierarchical network and the second hierarchical network are jointly trained.

[0023] When performing the joint training, the loss function is a weighted sum of a first loss function of the first hierarchical network and a second loss function of the second hierarchical network.

[0024] Preferably, when performing the joint training, the first loss function corresponding to each input business type is calculated based on the loss weight of all input labels; wherein the input business type is a business type to which an input label belongs.

[0025] When calculating the first loss function corresponding to any input business type, the loss weight of the input label of the any input business type is greater than the loss weight of the input label not belonging to the any input business type.

[0026] Preferably, the first loss function is a dice loss function, and the second loss function is a focal loss function.

[0027] Preferably, each state category under each current business type includes P standard images, and the P standard images include original images of different scene brightness, different shooting angles, and / or different postures under the each state category.

[0028] Preferably, the determination of the quality score result of the to-be-scored original image under the corresponding test business type comprises:

[0029] A weighted mean value is calculated for the R*P similarity results, and the calculation result is taken as the quality score result of the to-be-scored original image under the corresponding current business type.

[0030] The application also provides an image quality scoring method, comprising:

[0031] inputting an original image to be scored and an identifier of each current service type into a pre-trained quality scoring regression model to obtain a quality score result of the original image to be scored under each current service type;

[0032] The quality scoring regression model is a neural network model trained by using test sample images, an identifier of each test service type, and a quality score result of the test sample images under each test service type;

[0033] The determination process of the quality score result of the test sample images under each test service type comprises:

[0034] inputting the test sample images and labels corresponding to each test service type into a pre-trained state recognition model, marking image regions related to each test service type, and performing feature extraction based on the image regions to obtain a first state feature vector corresponding to each test service type;

[0035] For each first state feature vector, the similarity of the first state feature vector to a state feature vector of R*P standard images corresponding to the respective test service type is calculated to obtain R*P similarity results;

[0036] For each first state feature vector, the quality score result of the test sample images under the respective test service type is determined based on the R*P similarity results;

[0037] The state feature vector of the standard image is a state feature vector obtained by inputting a standard image and a label of the respective test service type into the state recognition model for feature extraction; the R*P standard images are typical images of unambiguous distinguishable state categories set for various states corresponding to the respective test service type, wherein R is the total number of all state categories under the respective test service type, and P is a positive integer.

[0038] Preferably, the state recognition model comprises a first hierarchical network for performing semantic segmentation and a second hierarchical network for performing state recognition;

[0039] The input of the first hierarchical network is the test sample images and the labels, and the output is a semantic feature map corresponding to each label, which is used to mark image regions related to each test service type;

[0040] The second hierarchical network comprises M sub-networks corresponding to M service types one by one; wherein M is the total number of all service types;

[0041] The input of each of the sub-networks is the test sample image and a modified semantic feature map corresponding to each label of the first test service type corresponding to the sub-network, each of the sub-networks connects the test sample image and the modified semantic feature map in a channel dimension, and inputs the connected image into a classification network for state recognition to obtain the first state feature vector corresponding to the first test service type; wherein the modified semantic feature map is a semantic feature map up-sampled to the same size as the test sample image.

[0042] Preferably, the inputting of the connected image into the classification network for state recognition comprises:

[0043] dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each of the two-dimensional slices in each of the two-dimensional slices;

[0044] inputting all the two-dimensional slices with added two-dimensional position information into the classification network for state recognition;

[0045] wherein, through training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current service type is ensured to be greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current service type.

[0046] Preferably, an attention mechanism is added in the classification network to assign a weight to each pixel input into the classification network, and the weight of the pixel located in the image region related to the test service type is greater than the weight of the pixel located outside the image region related to the test service type.

[0047] Preferably, in training the state recognition model, the first hierarchical network is trained using training sample images, and the second hierarchical network is trained based on the trained first hierarchical network; or,

[0048] In training the state recognition model, the first hierarchical network and the second hierarchical network are jointly trained; or,

[0049] In training the state recognition model, the first hierarchical network is initially trained using training sample images, the second hierarchical network is initially trained based on the first hierarchical network obtained after initial training, and the first hierarchical network and the second hierarchical network are jointly trained after initial training of the second hierarchical network;

[0050] When performing the joint training, a loss function is a weighted sum of a first loss function of the first hierarchical network and a second loss function of the second hierarchical network.

[0051] Preferably, when performing the joint training, the first loss function corresponding to each input service type is calculated based on loss weights of all input labels; wherein the input service type is a service type to which an input label belongs.

[0052] When calculating the first loss function corresponding to any input service type, the loss weight of the input label of the any input service type is greater than the loss weight of the input label not belonging to the any input service type.

[0053] Preferably, the determining of the quality score result of the test sample image under the corresponding test service type comprises:

[0054] A weighted mean of the R*P similarity results is calculated, and the calculation result is taken as the quality score result of the test sample image under the corresponding test service type.

[0055] The application provides an image quality scoring device, comprising a state recognition unit and a scoring unit.

[0056] The state recognition unit is configured to input an original image to be scored and a label corresponding to each current service type into a pre-trained state recognition model, mark an image region related to the each current service type, and perform feature extraction based on the image region to obtain a first state feature vector corresponding to the each current service type.

[0057] The scoring unit is configured to, for each first state feature vector, calculate a similarity between the first state feature vector and a state feature vector of R*P standard images corresponding to a corresponding test service type to obtain R*P similarity results; and further configured to, for each first state feature vector, determine a quality score result of the original image to be scored under a corresponding current service type based on the R*P similarity results.

[0058] The state feature vector of the standard image is a state feature vector obtained by inputting a standard image and a label of a corresponding current service type into the state recognition model for feature extraction; the R*P standard images are typical images of state categories that can be unambiguously distinguished, which are set for various states of the corresponding current service type, the R is a total number of all state categories under the corresponding current service type, and the P is a positive integer.

[0059] Preferably, in the state recognition unit, the state recognition model comprises a first hierarchical network for performing semantic segmentation and a second hierarchical network for performing state recognition.

[0060] the input of the first hierarchical network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type;

[0061] the second hierarchical network includes M sub-networks corresponding to M business types; wherein, M is the total number of all business types;

[0062] the input of each sub-network is the original image to be scored and the modified semantic feature map corresponding to each label of the first current business type corresponding to the sub-network, each sub-network connects the original image to be scored and the modified semantic feature map in the channel dimension, and inputs the connected image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current business type; wherein, the modified semantic feature map is a semantic feature map up-sampled to the same size as the original image to be scored.

[0063] Preferably, in the state recognition unit, the input of the connected image into the classification network for state recognition includes:

[0064] dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each two-dimensional slice in each two-dimensional slice;

[0065] inputting all two-dimensional slices with added two-dimensional position information into the classification network for state recognition;

[0066] wherein, through training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is ensured to be greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

[0067] Preferably, in the state recognition unit, an attention mechanism is added to the classification network to assign a weight to each pixel input into the classification network, and the weight of the pixel located in the image region related to the current business type is greater than the weight of the pixel located outside the image region related to the current business type.

[0068] Preferably, the device further comprises a training unit for training the state recognition model;

[0069] wherein, when the training unit trains the state recognition model, the training of the first hierarchical network is completed using training sample images, and the second hierarchical network is trained based on the trained first hierarchical network; or,

[0070] The first hierarchical network and the second hierarchical network are jointly trained when the training unit trains the state recognition model; or

[0071] The first hierarchical network is initially trained by using a training sample image when the training unit trains the state recognition model, the second hierarchical network is initially trained based on the first hierarchical network obtained after initial training, and the first hierarchical network and the second hierarchical network are jointly trained after initial training of the second hierarchical network.

[0072] The loss function is a weighted sum of a first loss function of the first hierarchical network and a second loss function of the second hierarchical network when the joint training is performed.

[0073] Preferably, the first loss function corresponding to each input service type is calculated based on the loss weight of all input labels when the joint training is performed, wherein the input service type is the service type to which the input label belongs.

[0074] The loss weight of the input label of any input service type is greater than the loss weight of the input label not belonging to the any input service type when the first loss function corresponding to the any input service type is calculated.

[0075] Preferably, in the state recognition unit, the determination of the quality score result of the to-be-scored original image under the corresponding current service type comprises:

[0076] The weighted mean of the R*P similarity results is calculated, and the calculation result is taken as the quality score result of the to-be-scored original image under the corresponding current service type.

[0077] The present application also provides an image quality scoring system, comprising a training device and an image quality scoring device;

[0078] The training device comprises a state recognition unit, a scoring unit and a first training unit.

[0079] The state recognition unit is configured to input a test sample image and a label corresponding to each test service type into a pre-trained state recognition model, mark an image region related to the each test service type, and perform feature extraction based on the image region to obtain a first state feature vector corresponding to the each test service type.

[0080] The scoring unit is configured to calculate, for each first state feature vector, a similarity between the first state feature vector and a state feature vector of R*P standard images corresponding to a respective test service type, to obtain R*P similarity results; and to determine, for each first state feature vector, a quality score result of the test sample image under the respective test service type based on the R*P similarity results.

[0081] The first training unit is configured to train a quality score regression model by using the test sample image, the identification of each current service type, and the quality score result of the test sample image under each current service type.

[0082] The image quality scoring device comprises an input unit, a quality score unit, and an output unit.

[0083] The input unit is configured to receive an original image to be scored and an identification of each current service type, and send the original image to be scored and the identification of each current service type to the quality score unit.

[0084] The quality score unit is configured to input the received original image to be scored and the identification of each current service type into the quality score regression model, to obtain a quality score result of the original image to be scored under each current service type.

[0085] The output unit is configured to output the quality score result of the original image to be scored under each current service type.

[0086] The state feature vector of the standard image is a state feature vector obtained by inputting a standard image and a label of a respective test service type into the state recognition model for feature extraction; the R*P standard images are typical images of unambiguous distinguishable state categories set for various states of the respective test service type, wherein R is a total number of all state categories under the respective test service type, and P is a positive integer.

[0087] Preferably, in the state recognition unit, the state recognition model comprises a first hierarchical network for semantic segmentation and a second hierarchical network for state recognition.

[0088] The input of the first hierarchical network is the test sample image and the label, and the output is a semantic feature map corresponding to each label, for marking an image region related to each test service type.

[0089] The second hierarchical network comprises M sub-networks corresponding to M service types one by one; wherein M is a total number of all service types.

[0090] The input of each of the sub-networks is the test sample image and a corrected semantic feature map corresponding to each label of the first test service type corresponding to the sub-network, each of the sub-networks connects the test sample image and the corrected semantic feature map in a channel dimension, and inputs the connected image into a classification network for state recognition to obtain the first state feature vector corresponding to the first test service type; wherein the corrected semantic feature map is a semantic feature map up-sampled to the same size as the test sample image.

[0091] Preferably, in the state recognition unit, the inputting of the connected image into the classification network for state recognition comprises:

[0092] dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each of the two-dimensional slices in each of the two-dimensional slices;

[0093] inputting all the two-dimensional slices with added two-dimensional position information into the classification network for state recognition;

[0094] wherein, through training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current service type is ensured to be greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current service type.

[0095] Preferably, in the state recognition unit, an attention mechanism is added to the classification network to assign a weight to each pixel input into the classification network, and the weight of a pixel located in the image region related to the test service type is greater than the weight of a pixel located outside the image region related to the test service type.

[0096] Preferably, the training device further comprises a second training unit for training the state recognition model.

[0097] wherein, when the state recognition model is trained in the second training unit, the training of the first hierarchical network is completed by using a training sample image, and the second hierarchical network is trained based on the trained first hierarchical network; or,

[0098] when the state recognition model is trained in the second training unit, the first hierarchical network and the second hierarchical network are jointly trained; or,

[0099] In the second training unit, the first hierarchical network is initially trained by using a training sample image, and the second hierarchical network is initially trained based on the first hierarchical network obtained after the initial training; and after the initial training of the second hierarchical network, the first hierarchical network and the second hierarchical network are jointly trained.

[0100] In the joint training, a loss function is a weighted sum of a first loss function of the first hierarchical network and a second loss function of the second hierarchical network.

[0101] Preferably, in the joint training of the second training unit, the first loss function corresponding to each input service type is calculated based on a loss weight of an input label; the input service type is a service type to which the input label belongs.

[0102] In the calculation of the first loss function corresponding to any input service type, the loss weight of the input label of the any input service type is greater than the loss weight of the input label not belonging to the any input service type.

[0103] Preferably, in the state recognition unit, the determination of the quality score result of the test sample image under the corresponding test service type comprises:

[0104] The weighted mean of the R*P similarity results is calculated, and the calculation result is taken as the quality score result of the test sample image under the corresponding test service type.

[0105] An image quality scoring device comprises an input unit, a quality scoring unit and an output unit.

[0106] The input unit is configured to receive an original image to be scored and an identification of each current service type, and send the original image to be scored and the identification of each current service type to the quality scoring unit.

[0107] The quality scoring unit is configured to input the received original image to be scored and the identification of each current service type into a quality scoring regression model pre-trained by a training device, to obtain a quality score result of the original image to be scored under each current service type.

[0108] The output unit is configured to output the quality score result of the original image to be scored under each current service type.

[0109] The training device comprises a state recognition unit, a scoring unit and a first training unit.

[0110] The state recognition unit is configured to input the test sample image and labels corresponding to each test service type into a pre-trained state recognition model, mark image regions related to each test service type, and perform feature extraction based on the image regions to obtain a first state feature vector corresponding to each test service type.

[0111] The scoring unit is configured to, for each first state feature vector, calculate a similarity between the first state feature vector and state feature vectors of R*P standard images corresponding to the respective test service type to obtain R*P similarity results, and determine a quality score result of the test sample image under the respective test service type based on the R*P similarity results.

[0112] The first training unit is configured to train the quality score regression model by using the test sample image, the identification of each current service type, and the quality score result of the test sample image under each current service type.

[0113] The state feature vector of the standard image is a state feature vector obtained by inputting a standard image and a label of a respective test service type into the state recognition model for feature extraction; the R*P standard images are typical images of state categories that can be unambiguously distinguished and are set for various states of the respective test service type, where R is a total number of all state categories under the respective test service type, and P is a positive integer.

[0114] Preferably, in the state recognition unit, the state recognition model includes a first hierarchical network for semantic segmentation and a second hierarchical network for state recognition.

[0115] The first hierarchical network takes the test sample image and the labels as input and outputs a semantic feature map corresponding to each label, which is used to mark image regions related to each test service type.

[0116] The second hierarchical network includes M sub-networks corresponding to M service types one by one, where M is a total number of all service types.

[0117] Each sub-network takes the test sample image and a corrected semantic feature map corresponding to each label of a first test service type corresponding to the sub-network as input, connects the test sample image and the corrected semantic feature map in a channel dimension, inputs the connected image into a classification network for state recognition to obtain a first state feature vector corresponding to the first test service type, and the corrected semantic feature map is a semantic feature map up-sampled to the same size as the test sample image.

[0118] Preferably, in the state recognition unit, the connected image is input into the classification network for state recognition, comprising:

[0119] The connected image is divided into a plurality of two-dimensional slices, and two-dimensional position information corresponding to each two-dimensional slice is added to each two-dimensional slice;

[0120] All two-dimensional slices with added two-dimensional position information are input into the classification network for state recognition;

[0121] Through training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

[0122] Preferably, in the state recognition unit, an attention mechanism is added to the classification network to assign weights to each pixel input into the classification network, and the weight of a pixel located in the image region related to the test business type is greater than the weight of a pixel located outside the image region related to the test business type.

[0123] Preferably, the training device further comprises a second training unit for training the state recognition model.

[0124] When the second training unit trains the state recognition model, the first-level network is trained using training sample images, and the second-level network is trained based on the trained first-level network; or

[0125] When the second training unit trains the state recognition model, the first-level network and the second-level network are jointly trained; or

[0126] When the second training unit trains the state recognition model, the first-level network is initially trained using training sample images, and the second-level network is initially trained based on the first-level network obtained after initial training; after initial training of the second-level network, the first-level network and the second-level network are jointly trained.

[0127] When the joint training is performed, the loss function is a weighted sum of a first loss function of the first-level network and a second loss function of the second-level network.

[0128] Preferably, when the second training unit performs the joint training, the first loss function corresponding to each input service type is calculated based on the loss weight of all input labels; wherein the input service type is the service type to which the input label belongs.

[0129] When the first loss function corresponding to any input service type is calculated, the loss weight of the input label of the any input service type is greater than the loss weight of the input label not belonging to the any input service type.

[0130] Preferably, in the state recognition unit, the determining of the quality score result of the test sample image under the corresponding test service type comprises:

[0131] A weighted mean is calculated for the R*P similarity results, and the calculation result is taken as the quality score result of the test sample image under the corresponding test service type.

[0132] The application also provides a training device, comprising: a state recognition unit, a scoring unit and a first training unit;

[0133] The state recognition unit is configured to input a test sample image and a label corresponding to each test service type into a pre-trained state recognition model, mark an image region related to the each test service type, and perform feature extraction based on the image region to obtain a first state feature vector corresponding to the each test service type;

[0134] The scoring unit is configured to, for each first state feature vector, calculate the similarity between the first state feature vector and a state feature vector of R*P standard images corresponding to the corresponding test service type to obtain R*P similarity results; and further configured to, for each first state feature vector, determine a quality score result of the test sample image under the corresponding test service type based on the R*P similarity results.

[0135] The first training unit is configured to train a quality score regression model by using the test sample image, the identification of each current service type, and the quality score result of the test sample image under the each current service type.

[0136] The quality score model is used to process an inputted original image to be scored and an identification of each current service type, to obtain a quality score result of the original image to be scored under each current service type; the state feature vector of the standard image is a state feature vector obtained after a standard image and a corresponding test service type label are input into the state recognition model for feature extraction; the R*P standard images are typical images of unambiguous distinguishable state categories set for various states of the corresponding test service type, R is a total number of all state categories under the corresponding test service type, and P is a positive integer.

[0137] Preferably, in the state recognition unit, the state recognition model comprises a first level network for semantic segmentation and a second level network for state recognition.

[0138] The input of the first level network is the test sample image and the label, and the output is a semantic feature map corresponding to each label, used for marking an image region related to each test service type.

[0139] The second level network comprises M sub-networks corresponding to M service types one by one; M is a total number of all service types.

[0140] The input of each sub-network is the test sample image and a corrected semantic feature map corresponding to each label of a first test service type corresponding to the sub-network, each sub-network connects the test sample image and the corrected semantic feature map in a channel dimension, inputs the connected image into a classification network for state recognition, and obtains the first state feature vector corresponding to the first test service type; the corrected semantic feature map is a semantic feature map up-sampled to the same size as the test sample image.

[0141] Preferably, in the state recognition unit, the inputting the connected image into the classification network for state recognition comprises:

[0142] Dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each two-dimensional slice in each two-dimensional slice;

[0143] Inputting all two-dimensional slices with added two-dimensional position information into the classification network for state recognition.

[0144] Through training of the state recognition model, the weight of two-dimensional position information corresponding to a two-dimensional slice located in the image region related to the current service type is ensured to be greater than the weight of two-dimensional position information corresponding to a two-dimensional slice located outside the image region related to the current service type.

[0145] Preferably, in the state recognition unit, an attention mechanism is added to the classification network, and each pixel input into the classification network is assigned a weight, and the weight of a pixel located in the image region related to the test service type is greater than the weight of a pixel located outside the image region related to the test service type.

[0146] Preferably, the training device further comprises a second training unit for training the state recognition model.

[0147] In the training of the state recognition model by the second training unit, the first hierarchical network is trained by using training sample images, and the second hierarchical network is trained based on the trained first hierarchical network; or,

[0148] In the training of the state recognition model by the second training unit, the first hierarchical network and the second hierarchical network are jointly trained; or,

[0149] In the training of the state recognition model by the second training unit, the first hierarchical network is initially trained by using training sample images, and the second hierarchical network is initially trained based on the first hierarchical network obtained after the initial training; and after the initial training of the second hierarchical network, the first hierarchical network and the second hierarchical network are jointly trained.

[0150] In the joint training, the loss function is a weighted sum of a first loss function of the first hierarchical network and a second loss function of the second hierarchical network.

[0151] Preferably, in the joint training by the second training unit, the first loss function corresponding to each input service type is calculated based on the loss weight of all input labels, wherein the input service type is the service type to which the input label belongs.

[0152] In the calculation of the first loss function corresponding to any input service type, the loss weight of the input label of the any input service type is greater than the loss weight of the input label not belonging to the any input service type.

[0153] Preferably, in the state recognition unit, the determination of the quality score result of the test sample image under the corresponding test service type comprises:

[0154] The weighted mean of the R*P similarity results is calculated, and the calculation result is taken as the quality score result of the test sample image under the corresponding test service type.

[0155] The application provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to implement the image quality scoring method.

[0156] The application provides an electronic device, which comprises at least a computer readable storage medium and a processor.

[0157] The processor is configured to read the executable instructions from the computer readable storage medium and execute the instructions to implement the image quality scoring method.

[0158] According to the above technical solution, the pre-trained state recognition model can be used to process the input original image and the label corresponding to the business type, mark the image area related to each business type, extract the features based on the image area to obtain the first state feature vector, so that the first state feature vector obtained by processing mainly reflects the feature information of the region related to the business type; next, the R*P standard images corresponding to the business type are input into the state recognition model to obtain the state feature vectors of the R*P standard images, and the similarity of the first state feature vector and the state feature vectors of the R*P standard images is calculated to determine the quality score of the original image under a certain business type. When performing image quality scoring, the quality score of the image under the business type can be directly obtained by the above image quality scoring process, or a score regression network can be used to fit the above image quality scoring process. In actual image scoring, the image to be scored and the business type identifier are input into the trained score regression network to obtain the quality score of the image under the corresponding business type.

[0159] Through the above processing, the current recognized business type is used as supervision information, so that the first state feature vector used for similarity comparison mainly reflects the feature information of the region related to the business type, and the image quality judgment performed by the first state feature vector pays more attention to the region related to the business, rather than the whole image quality judgment, so that the image quality judgment can be reasonably performed according to different business types, and the quality judgment result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0160] Figure 1 It is a basic flowchart of the first image quality scoring method in the application;

[0161] Figure 2 It is a specific flowchart of the image quality scoring method of the first embodiment of the application;

[0162] Figure 3 It is a schematic diagram of the overall framework of the illegal business judgment model in the first embodiment;

[0163] Figure 4 Structure diagram of each sub-network in the second level network of embodiment one;

[0164] Figure 5 Specific flow diagram of the image quality scoring method in embodiment two;

[0165] Figure 6 Basic structure diagram of the first image quality scoring device in the present application;

[0166] Figure 7 Basic structure diagram of the image quality scoring system corresponding to the second image quality scoring method in the present application;

[0167] Figure 8 Basic structure diagram of the second image quality scoring device in the present application;

[0168] Figure 9 Basic structure diagram of the training device in the image quality scoring system of the present application;

[0169] Figure 10 Basic structure diagram of the electronic device provided in the present application. DETAILED DESCRIPTION

[0170] In order to make the purposes, technical means and advantages of the present application more clear, the present application is further described in detail below with reference to the accompanying drawings.

[0171] As described in the background section, before state recognition, image quality usually needs to be scored to filter out images with poor quality and improve the accuracy of state recognition. In application scenarios with multiple different recognition services, existing image quality scoring methods need manual calibration for scoring, and some cannot perform targeted image quality scoring for different recognition services, resulting in low accuracy of image quality scoring results.

[0172] For example, in the application of driving cabin illegal behavior judgment, illegal behavior involves a wide range of different illegal behavior judgment services (such as judgment services of whether to wear a safety belt, whether to make a phone call, whether to smoke, whether to hold a baby, etc.); At the same time, in the application of driving cabin illegal behavior judgment, more areas need to be concerned (such as the main driver, the co-driver, the rear seat passenger, etc.); Different judgment services focus on different areas, and image quality scoring will affect the accuracy of the scoring results if it is generalized; If different image quality scoring methods are maintained for different illegal behavior judgment services, the version is multiple and difficult to maintain.

[0173] Based on the above analysis, the present application provides an image quality scoring method suitable for various different identification services, the basic idea of which is that by setting labels corresponding to different identification services (hereinafter referred to as services), a trained state recognition model is used to perform semantic segmentation on the input original image, mark the image area related to the current service type, and perform image feature extraction based on the image area for image quality scoring. In this way, the state recognition model is driven by the service to focus on the area related to the current identification service type, and the image quality is judged based on the relevant area instead of the whole image, so that the image quality is reasonably judged corresponding to different identification service types, and the image quality scoring result is more accurate.

[0174] Based on the basic idea of the present application, the present application provides two image quality scoring methods. The basic flow of the first method is as shown in Figure 1

[0175] Step 101, input the original image to be scored and the label corresponding to each current service type into the pre-trained state recognition model, mark the image area related to each current service type, and perform feature extraction based on the image area to obtain a first state feature vector corresponding to each current service type.

[0176] The state recognition model in the present application is a pre-trained model that can recognize the current state based on the original image. For example, in the application of driving cabin illegal behavior judgment, the state recognition model can be a driving cabin behavior judgment model, which judges the current state as not wearing a seat belt, making a phone call, or holding a baby, etc.

[0177] Since the state recognition model in the present application is trained and processed with service type as supervision information, it is necessary to set corresponding labels in advance for various to-be-identified service types, and one service type can correspond to one or more labels. For different service types, the number of corresponding labels can be the same or different. Preferably, the label can correspond to the area related to the service type. For example, in the application of driving cabin illegal behavior judgment, the illegal service type (i.e. the to-be-identified service type) can include: seat belt service, phone call service, baby holding service, etc. For the seat belt service, two labels can be set, which are trunk and seat belt, corresponding to the trunk-related area and the seat belt-related area.

[0178] ​In the processing of the present step, the original image to be scored and the label corresponding to each current business type are taken as the input of the state recognition model, and the original image is subjected to semantic segmentation by using the pre-trained state recognition model to obtain the image region corresponding to the input label, that is, the image region related to each current business type, which can be a mask graph corresponding to the input label, for example.

[0179] By using the state recognition model, after determining the image region related to each current business type, the image classification feature of the relevant region is extracted based on the image corresponding to each current business type to obtain a state feature vector, and the state feature vector corresponds to a certain current business type. The obtained state feature vector is referred to as a first state feature vector. Since the first state feature vector is obtained based on the image region related to a certain current business type, the first state feature vector mainly reflects the feature information of the region related to a certain current business type.

[0180] The current business type is the business type to be recognized in the present processing, for example, the state recognition model inputs an original image and a label corresponding to the safety belt business in the present processing, and then a first state feature vector corresponding to the safety belt business can be obtained by the processing of the state recognition model, which can be used to determine which state the safety belt business is in. In addition, the current business type can be one or more, for example, an original image and labels corresponding to the safety belt business and the telephone business can be input in one processing, and then a first state feature vector corresponding to the safety belt business and a first state feature vector corresponding to the telephone business can be obtained by the processing of the state recognition model, which can be used to determine which state the safety belt business is in and which state the telephone business is in, respectively.

[0181] In step 102, for each first state feature vector, the similarity between the first state feature vector and the state feature vector of the R*P standard images corresponding to the corresponding current business type is calculated to obtain R*P similarity results.

[0182] As described above, the first state feature vector is a feature vector obtained by processing the input original image by using the state recognition model.

[0183] In the present application, in order to correspond to different business types for image quality scoring, R*P standard images are set for each business type, and the R*P standard images are input into the state recognition model to obtain the state feature vectors of the R*P standard images. The R*P standard images are typical images that can unambiguously distinguish the state categories corresponding to a certain business type, R is the total number of all state categories under a certain business type, and P is a positive integer.

[0184] Preferably, the P standard images set for each state category corresponding to the business type include original images of different scene brightness, different shooting angles and / or different postures under the corresponding state category.

[0185] The R*P standard images can unambiguously determine the state category, and the feature information of the standard image obtained by using the state recognition model represents the feature of the image that can unambiguously determine the state category; for the first state feature information corresponding to each current business type B, the similarity between the first state feature information and the state feature information of the standard image corresponding to the business type B is calculated respectively, and R*P similarity results are obtained, and the similarity results can reflect the similarity between the first state feature information and the image feature that can unambiguously determine the state category.

[0186] In step 103, for each first state feature vector, based on the R*P similarity results, the quality score result of the original image to be scored under the corresponding current business type is determined.

[0187] Since the R*P similarity results determined in step 102 can reflect the similarity between the first state feature information and the image feature that can unambiguously determine the state category, the quality of the original image can be scored using these similarity results to reflect the quality of the original image relative to the standard image, thereby effectively giving a reasonable quality score of the image for each current business type.

[0188] So far, Figure 1 The method flow shown ends.

[0189] In addition, in order to further optimize the processing delay and resource occupation of the image quality scoring, the above first method can also be regarded as a complete image quality scoring process, and a quality score regression model is trained to fit the image quality scoring process given by the first method, that is, a series of test sample images and test business types are executed by the image quality scoring process given by the first method, and the quality score results of the test sample images under the test business types are obtained., and then the test sample images and the corresponding quality score results are used as training samples to train and generate a quality score regression model to fit the image quality scoring process given by the first method. The second image quality scoring method provided in the present application is based on the trained quality score regression model to score the image quality. Specifically, the second image quality scoring method comprises:

[0190] inputting the original image to be scored and the identifier of each current business type into the pre-trained quality score regression model to obtain the quality score result of the original image to be scored under each current business type;

[0191] Among them, the quality score regression model is a neural network model trained using test sample images, the identifier of each test business type, and the quality score results of the test sample images under each test business type;

[0192] The process for determining the quality score of test sample images for each test service type includes:

[0193] The test sample images and labels corresponding to each test business type are input into the pre-trained state recognition model to mark the image regions related to each test business type, and feature extraction is performed based on the image regions to obtain the first state feature vector corresponding to each test business type.

[0194] For each first state feature vector, calculate the similarity between the first state feature vector and the state feature vectors of the corresponding R*P standard images of the test service type, and obtain R*P similarity results;

[0195] For each first-state feature vector, based on R*P similarity results, the quality score of the test sample image under the corresponding test service type is determined.

[0196] Both of the aforementioned image quality scoring methods introduce a state feature vector of a standard image, which is compared with the first state feature vector of the input image to obtain a similarity score, thus determining the image quality score. The state feature vector of the standard image refers to the state feature vector obtained after feature extraction from the standard image and the label input state recognition model for the corresponding test business type. For each business type, R*P standard images are provided. These standard images are typical images that unambiguously distinguish the state categories for various states corresponding to a business type, where R is the total number of all state categories under a certain business type, and P is a positive integer, representing the number of typical images for each state. For example, in the application scenario of judging illegal behavior in the driver's cab, the business type is judging whether a phone call has been made. R can be the total number of state categories under this business, and specific states can be such as holding the phone while making a call, holding the phone close to the ear while making a call, not making a call, etc. For each state category, P typical images are provided, such as typical images that take into account different scene brightness, various typical postures of the driver in the cab, and different shooting angles.

[0197] In the two image quality scoring methods mentioned above, the first state feature vector used for similarity comparison mainly reflects the feature information of the area related to the business type. Therefore, the image quality judgment focuses more on the area related to the business rather than judging the quality of the entire image. This allows for targeted and reasonable image quality judgment for different business types, making the quality judgment results more accurate.

[0198] The two image quality scoring methods of the present application are described in detail below through two specific embodiments.

[0199] Embodiment One

[0200] This embodiment is used to introduce the first image quality scoring method in detail.

[0201] Figure 2 A specific embodiment flowchart of the first image quality scoring method in the present application is shown in FIG. 1. For the convenience of description, the application scenario of the cab judging illegal behavior is taken as an example to illustrate the image quality scoring method in this specific embodiment, and the training process of the state recognition model is also described. As shown in FIG. 1, the method comprises the following steps. Figure 2

[0202] Step 201, training the illegal service judgment model by using training sample images.

[0203] The illegal service judgment model in this embodiment is the aforementioned state recognition model.

[0204] The illegal service judgment model in this embodiment is trained and processed with illegal service types as supervision information. Therefore, corresponding labels need to be set in advance for various illegal service types, and one illegal service type may correspond to one or more labels. For different illegal service types, the number of corresponding labels may be the same or different. Preferably, the labels can correspond to regions related to the illegal service types. For example, for the seat belt service, two labels can be set, which are trunk and seat belt, corresponding to the trunk-related region and the seat belt-related region.

[0205] The input of the illegal service judgment model includes the original image used for quality scoring and the label corresponding to one or more illegal service types (hereinafter referred to as input illegal type), and the illegal service judgment model is used to mark the image region related to each input illegal type in the original image, and then based on the marked image region, the classification feature is extracted to obtain the state feature vector corresponding to each input illegal type for output.

[0206] The illegal service judgment model described above can be implemented by using various network structures. This embodiment gives a specific implementation structure.

[0207] The illegal service judgment model in this embodiment is a neural network model including two hierarchical networks, the first hierarchical network is a semantic segmentation network, and the second hierarchical network is a semantic fusion and service recognition network. The overall framework of the illegal service judgment model in this embodiment can be as shown in FIG. 2. Figure 3

[0208] First, introduce the first hierarchical network: ​​

[0209] 1. In the first-level semantic segmentation network, the selection of semantics is determined by the type of illegal business. In fact, semantics can directly correspond to the input labels, and semantic segmentation is to process according to the input labels.

[0210] 2. The first-level network can be any neural network model used for image processing, such as... Figure 3 The convolutional neural network shown is a CNN network;

[0211] 3. When performing semantic segmentation on the input original image, first distinguish and mark the regions corresponding to each input label in the input original image to obtain a Mask image;

[0212] Since the input labels may correspond to one or more types of illegal business operations, the Mask image distinguishes and marks the label regions corresponding to all types of illegal business operations, that is, it marks the regions associated with each type of illegal business operation. This may include image regions associated with one or more types of illegal business operations. For example, assuming the input labels include labels for seatbelt services and telephone services (specifically including the head, arm, and hand), that is, including the torso, seatbelt, head, arm, and hand, in the semantic segmentation network, the regions corresponding to the torso, seatbelt, head, arm, and hand are first marked to obtain the Mask image;

[0213] 4. Based on the Mask map obtained in point 3, for each input label A, only the label of the region corresponding to the input label is retained to generate the Mask map corresponding to label A; in this way, for each input label, a corresponding Mask map is obtained, and the Mask maps corresponding to all input labels are the output of the semantic segmentation network. The Mask map corresponding to the input label is also called the semantic feature map.

[0214] As can be seen from the above, through the processing of the semantic segmentation network, a semantic feature map corresponding to each input label is obtained. The semantic feature map corresponding to the label belonging to the same type of illegal business marks the image region related to that type of illegal business.

[0215] Next, we will introduce the second-level network:

[0216] 1. Second-level network such as Figure 3As shown, M sub-networks corresponding to M service types are included, and the structure of each sub-network is the same; M is the total number of all service types to be identified. The first-level network outputs semantic feature maps corresponding to each input label, and after entering the second-level network, the semantic feature maps corresponding to the labels of the same input service type and the original image (i.e., the original image input into the first-level network) are input into the sub-network corresponding to the corresponding input service type. For example, the second-level network includes three sub-networks corresponding to the seat belt service, the phone call service and the baby holding service; the semantic feature maps output by the first-level network are corresponding to the labels of the seat belt service and the labels of the phone call service, so when entering the second-level network, all semantic feature maps corresponding to the labels of the seat belt service and the original image are input into the sub-network corresponding to the seat belt service, and all semantic feature maps corresponding to the labels of the phone call service and the original image are input into the sub-network corresponding to the phone call service. The following description is described by taking one sub-network and the service type X corresponding to the sub-network as an example;

[0217] 2. The processing of the sub-network includes two parts: fusion processing and a classification network, as shown in detail in Figure 4

[0218] 3. Fusion processing:

[0219] All semantic feature maps input into the sub-network are corrected, i.e., up-sampled to the same size as the original image; the corrected all semantic feature maps and the original image are concatenated, i.e., connected in the channel dimension, and output to the classification network as the fusion processing result;

[0220] 4. Classification network:

[0221] Most simply, the concatenated image obtained by the fusion processing can be input into the classification network for image classification feature extraction and classification result determination, to obtain the state feature vector corresponding to the service type X and the state classification result. For example, for the sub-network corresponding to the seat belt service, the state feature vector corresponding to the seat belt service and the state classification result of the seat belt service (such as not wearing a seat belt) are finally obtained. The network structure of the classification network can adopt various existing neural network models for realizing classification, such as a CNN network.

[0222] In addition, in the classification network processing, to further accelerate the convergence of the classification task and improve the robustness of the classification network, the concatenated image output by the fusion processing can be optionally divided into a plurality of two-dimensional patches, assuming that the size of the concatenated image is H*W*C, H and W are the height and width of the image respectively, C is the number of channels of the concatenated image, and the size of each two-dimensional patch is P*P, then after the concatenated image is divided into two-dimensional patches, the number of two-dimensional patches is H*W*C / P 2 ​In each two-dimensional slice, the two-dimensional position information corresponding to the two-dimensional slice is added; and all the two-dimensional slices with the added two-dimensional position information are input into a classification network for state recognition.

[0223] Through training of the state recognition model, the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the business type X has a greater weight than the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the business type X. Here, the process of adding the two-dimensional position information to the two-dimensional slice can be a dot multiplication operation of the two-dimensional slice and the two-dimensional position information weight. For example, when the business type X is a safety belt business, through training, the two-dimensional position information weight (which can be 2, for example) corresponding to the two-dimensional slice located in the safety belt related region is greater than the two-dimensional position information weight (which can be 1, for example) corresponding to the two-dimensional slice located in the safety belt related region.

[0224] In the classification network processing, to further improve the classification accuracy of the classification network, an attention mechanism can be optionally introduced into the classification network to assign a weight to each pixel input into the classification network, and through training, the weight of the pixel located in the image region related to the business type X is greater than the weight of the pixel located outside the image region related to the business type X. In this way, more critical and important feature information can be extracted to enable the classification network to make more accurate classification result judgments, while not causing more consumption of the calculation and storage of the state recognition model.

[0225] The above is an exemplary architecture and specific process of the illegal business judgment model given in this embodiment.

[0226] When training the illegal business judgment model (i.e., the state recognition model), a training sample image and one or more labels corresponding to training business types are input, the processing result is compared with the known state category of the training sample image to obtain a loss function, and the parameters of the illegal business judgment model are adjusted according to the loss function.

[0227] Specifically, for the illegal business judgment model composed of two hierarchical networks, the two hierarchical networks can be trained separately, or the two hierarchical networks can be jointly trained, or the two hierarchical networks can be trained in stages, i.e., the two hierarchical networks are trained separately in the initial stage, and then the two hierarchical networks are jointly trained after the initial stage.

[0228] Specifically, when training the two hierarchical networks respectively, a part of the training sample images can be used for training the first hierarchical network, the parameters of the first hierarchical network are adjusted according to the loss function of the first hierarchical network (hereinafter referred to as the first loss function), and then the first hierarchical network with updated parameters is used to process the training sample images until the training end condition of the first hierarchical network is reached; after the first hierarchical network is trained, another part of the training sample images is used to train the second hierarchical network, and the first hierarchical network is also needed when the second hierarchical network is trained. The first hierarchical network used here has been trained and the parameter values have been fixed. After processing by the second hierarchical network, the parameters of the second hierarchical network are adjusted according to the loss function of the second hierarchical network (hereinafter referred to as the second loss function), and then the first hierarchical network and the second hierarchical network with updated parameters are used to process the training sample images until the training end condition of the second hierarchical network is reached.

[0229] When the first hierarchical network is trained alone, the input label can correspond to one or more service types. For example, if the input label includes all N labels corresponding to all service types (N is the total number of all labels corresponding to all service types), the first loss function can be the result of comparing the Mask graph obtained by semantic segmentation, which labels all N label regions, with the standard Mask graph of the training sample image (where the N label regions are accurately labeled). Generally, the same label regions can be compared one by one, and the final Mask graph comparison result can be obtained as the first loss function according to the comparison result of each region. The weight of the comparison result of different label regions in the final result calculation is usually the same.

[0230] When the second hierarchical network is trained alone, since the second hierarchical network includes M sub-networks, the second loss function corresponding to each sub-network is determined according to the processing result of the sub-network during training, and is used to adjust the parameters of the corresponding sub-network. Specifically, the training sample image is first input into the trained first hierarchical network, and after processing by the first hierarchical network, the semantic feature map corresponding to each input label and the training sample image are input into the second hierarchical network; the second hierarchical network inputs the semantic feature map corresponding to the input label belonging to the same service type X and the training sample image into the sub-network Y corresponding to the service type X, determines the classification of the training sample image under the service type X after processing by the sub-network Y, compares it with the standard classification of the training sample image under the service type to determine the second loss function, and adjusts the parameters of the sub-network Y according to the second loss function.

[0231] When the two hierarchical networks are jointly trained, the training sample image can be input into the first hierarchical network, and after processing by the first hierarchical network, a first loss function is obtained. Then, the processing result of the first hierarchical network is input into the second hierarchical network according to the input requirement of the second hierarchical network, and after processing by the second hierarchical network, a second loss function is obtained. Then, a joint loss function is obtained according to the weighted sum of the first loss function and the second loss function. The parameters of the first hierarchical network and the second hierarchical network are adjusted according to the joint loss function. Then, the updated first hierarchical network and the second hierarchical network process the input training sample image until the training end condition of the illegal business judgment model is reached.

[0232] In the joint training, the input label can be one or more business type labels. Since joint training is performed, the first loss function and the second loss function need to be weighted and summed to determine the final joint loss function. The second loss function corresponds to a single business type, so the first loss function also needs to be calculated for a single business type. Specifically, after the training sample image and one or more business type labels are input into the first hierarchical network, the processing is performed to obtain a Mask graph marked with the regions corresponding to all input labels. The Mask graph is compared with a standard Mask graph in which the regions corresponding to the input labels in the training sample image are marked. For each business type, the first loss function corresponding to the single business type is calculated. The determination method of the first loss function corresponding to a single business type is described below.

[0233] Assuming that the input labels correspond to two business types, namely safety belt business and phone call business, the Mask map can be obtained in the manner described in the previous paragraph, the Mask map marks the label regions of the safety belt business and the phone call business, the Mask map is compared with the standard Mask map, specifically, the regions corresponding to the same input labels in the Mask map and the standard Mask map are compared, and weights are set for each input label, and the first loss function S1 corresponding to the safety belt business and the first loss function S2 corresponding to the phone call business are determined according to the comparison results of each input label region and the weights of the input labels. When calculating the first loss function corresponding to the safety belt business, the weight of the label corresponding to the safety belt business (i.e., the torso and the safety belt) is greater than the weight of the label corresponding to the phone call business (i.e., the head, the arm and the palm); when calculating the first loss function corresponding to the phone call business, the weight of the label corresponding to the phone call business (i.e., the head, the arm and the palm) is greater than the weight of the label corresponding to the safety belt business (i.e., the torso and the safety belt). That is, when performing joint training, the first loss function of each input business type is calculated based on the loss weights of all input labels, and when calculating the first loss function corresponding to a certain input business type, the loss weight of the label belonging to the input business type is greater than the loss weight of the label not belonging to the input business type. The above processing can ensure that the label region matching the target business has a greater impact on the first loss function, so that the model pays more attention to the region related to the target business.

[0234] When training the two hierarchical networks in stages, the first hierarchical network can be initially trained using the training sample images, and the second hierarchical network can be initially trained based on the first hierarchical network obtained after the initial training; after the initial training of the first hierarchical network and the second hierarchical network is completed, the first hierarchical network and the second hierarchical network are jointly trained.

[0235] The first loss function can be various loss functions for measuring distance loss, such as a Dice loss function, and the second loss function can be various loss functions for measuring classification loss, such as a Focal loss function or a cross-entropy loss function, etc.

[0236] As described above, the illegal business judgment model can be trained, and the trained illegal business judgment model is used to process the original image to be scored through steps 202 and 203.

[0237] In step 202, the original image to be scored and the label corresponding to each current business type are input into the pre-trained illegal business judgment model, and the image region related to each current business type is marked.

[0238] As can be seen from the description of the illegal business judgment model in step 201, the trained illegal business judgment model can determine the state of one or more business types for an original image.

[0239] When it is actually needed to perform image quality scoring of an original image under one or more business types, the original image to be scored and the label corresponding to each target business type (hereinafter referred to as the current business type) for which quality scoring is needed can be input into the illegal business model to mark the image area related to each current business type.

[0240] When the foregoing two-level network is used to implement the illegal business model, the processing of the present step is also the processing of the first-level network, that is, semantic segmentation of the original image is performed to obtain a Mask graph corresponding to each input label.

[0241] Step 203, in the illegal business judgment model, feature extraction is performed based on the image area related to each current business type to obtain a first state feature vector corresponding to each current business type.

[0242] For each current business type, after the image area related to the current business type is marked in step 202, image classification feature extraction is performed on the original image based on the marked image area to obtain a first state feature vector corresponding to the corresponding business type.

[0243] When the foregoing two-level network is used to implement the illegal business model, the processing of the present step is also the processing of the second-level network, that is, for the i-th current business type, based on the Mask graph corresponding to the label of the corresponding business type and the original image, fusion processing and feature extraction processing of the classification network are performed to obtain a classification feature vector of the fused image, that is, a first state feature vector corresponding to the i-th current business type. Wherein, i is the index of the current business type.

[0244] Next, steps 204 and 205 are performed on each first state feature vector to obtain a quality score result of the original image under the business type corresponding to the first state feature vector.

[0245] Step 204, for each first state feature vector, the similarity of the first state feature vector with the state feature vectors of R*P standard images of the corresponding current business type is calculated to obtain R*P similarity results.

[0246] The processing of each first state feature vector in the present step is the same, and for the sake of convenience, the processing of the first state feature vector corresponding to a current business type X is taken as an example to illustrate the specific processing of the present step.

[0247] As described above, in the present application, in order to correspond to different service types for image quality scoring, the corresponding R*P standard images are set for each service type. In this step, the R*P standard images corresponding to the current service type X and the state feature vectors of the R*P standard images are determined. Wherein, each standard image and the label of the current service type X are input into the illegal service judgment model, and after the processing of the illegal service judgment model, the output classification feature vector, that is, the state feature vector of the standard image, is obtained. The processing of the standard image to obtain the state feature vector of the standard image can be performed in this step, or it can be performed in advance, and the state feature vector of the standard image obtained after the prediction processing is obtained in this step.

[0248] The first state feature vector is compared with the state feature vectors of the R*P standard images corresponding to the current service type X respectively, and R*P similarity results corresponding to the current service type X are obtained.

[0249] In step 205, for each first state feature vector, based on the R*P similarity results, the quality score result of the original image to be scored under the corresponding current service type is determined.

[0250] The processing of each first state feature vector in this step is the same, and for the convenience of description, the processing of a first state feature vector corresponding to a current service type X is taken as an example to illustrate the specific processing of this step.

[0251] For the R*P similarity results corresponding to the current service type X, the R*P similarity results are used to calculate the quality score result of the original image to be scored under the current service type X. The specific calculation method can be to calculate the weighted average value of the R*P similarity results, and of course the calculation method of the weighted average value is only an example, and the actual application is not limited to this calculation method.

[0252] Since the R*P similarity results determined in step 204 can reflect the similarity between the first state feature information and the image features of the standard image (i.e. the image that can unambiguously distinguish the state), the quality of the original image can be scored by using these similarity results to reflect the quality of the original image relative to the standard image, so as to effectively give a reasonable quality score of the image for each current service type.

[0253] So far, Figure 2 The specific embodiment flow of the first image quality scoring method shown ends.

[0254] Embodiment two:

[0255] This embodiment is used to introduce the second image quality scoring method in detail.

[0256] Figure 5 The second embodiment of the image quality scoring method in the present application is shown in the flowchart. For the convenience of description, the application scenario of the cab judging illegal behavior is still taken as an example to illustrate the image quality scoring method in the embodiment, and the training process of the illegal behavior judgment model and the quality scoring regression model is described together. As shown in Figure 5 The method comprises the following steps.

[0257] Step 501, training the illegal behavior judgment model by using the training sample images.

[0258] The processing of this step is the same as that of step 201, which will not be described here.

[0259] Step 502, inputting the test sample images and the labels corresponding to each test business type into the pre-trained illegal behavior judgment model to mark the image regions related to each test business type.

[0260] The test sample images and the test business types in the embodiment are the sample images used for training the quality scoring regression model. In the processing of the illegal behavior judgment model, the processing of the test sample images and the test business types is the same as the processing of the original image to be scored and the current business type in the first embodiment, respectively. Only in the embodiment, the test sample images are processed to obtain the corresponding image quality scores, and the purpose is to use the test sample images and their image quality score results as training samples to train the quality scoring regression model.

[0261] Since the processing of the test sample images and the test business types is the same as the processing of the original image to be scored and the current business type in the first embodiment, respectively, the detailed processing process of steps 502-505 will not be described here.

[0262] Step 503, performing feature extraction based on the image regions related to each test business type in the illegal behavior judgment model to obtain the first state feature vector corresponding to each test business type.

[0263] Step 504, for each first state feature vector, calculating the similarity between the first state feature vector and the state feature vectors of the R*P standard images of the corresponding test business type to obtain R*P similarity results.

[0264] Step 505, for each first state feature vector, determining the quality score result of the test sample image under the corresponding test business type based on the R*P similarity results.

[0265] By repeatedly executing steps 502-505 on multiple test sample images, multiple test sample images and their corresponding quality score results are obtained.

[0266] Step 506: Use multiple test sample images and their corresponding quality score results as training samples to train and generate a quality score regression model.

[0267] The test sample images may have one or more quality scores, corresponding to one or more test service types. The network structure of the quality score regression model can use existing neural network models, such as CNN networks.

[0268] When training the quality scoring regression model, the test sample image and the identifier of the test service type are taken as input. The output is the quality score result of the test sample image under the test service type. This quality score result is compared with the quality score result obtained in step 505 to obtain the loss function. The parameters of the quality scoring regression model are then adjusted according to the loss function. The input test service type identifier can be one or more identifiers. In other words, the trained quality scoring regression model can obtain image quality scores for one or more service types for a single original image. The quality scoring regression model trained in this way can fit the image quality scoring process in steps 502-505 as closely as possible.

[0269] Step 507: When it is necessary to perform image quality scoring on the original image to be scored, input the original image to be scored and the identifier of each current business type into the pre-trained quality scoring regression model to obtain the quality score result of the original image to be scored under each current business type.

[0270] The aforementioned steps 501-506 can all be categorized as training the quality scoring regression model. Since this part of the process is time-consuming and resource-intensive, it can be pre-processed. When the original image to be scored is actually needed, step 507 is executed directly to complete the real-time image quality scoring. In this way, on the one hand, the pre-processed training significantly saves time and resources during real-time scoring; on the other hand, by fitting the quality scoring regression model to the image quality scoring process of steps 502-506, the image quality scoring results are ensured to be applicable to various business types, improving the accuracy of image quality scoring, and simultaneously enabling image quality scoring for multiple business types.

[0271] At this point, Figure 5 The process for the specific embodiment of the second image quality scoring method shown here has ended.

[0272] The above is the specific implementation of the two image quality scoring methods provided in the present application. In the present application, the score of the image quality is generated by weak supervision of the business type, not by appearance similarity, template comparison, or human visual perception, but by business driving, training a state recognition model, calculating the average similarity of the features of the original image and multiple standard samples of unambiguous distinguishable states, and taking the average similarity as the quality score, which fully serves the needs of the state recognition business and improves the accuracy of the image quality scoring.

[0273] The present application corresponding to the above two image quality scoring methods also provides two image quality scoring devices. Figure 6 For the first image quality scoring device, the first image quality scoring method can be implemented. As shown in Figure 6 The first image quality scoring device includes a state recognition unit and a scoring unit.

[0274] The state recognition unit is configured to input the original image to be scored and the labels corresponding to each current business type into a pre-trained state recognition model, mark the image regions related to each current business type, and perform feature extraction based on the image regions to obtain a first state feature vector corresponding to each current business type.

[0275] The scoring unit is configured to, for each first state feature vector, calculate the similarity of the first state feature vector with the state feature vectors of R*P standard images corresponding to the corresponding test business type, to obtain R*P similarity results; and to, for each first state feature vector, determine the quality score of the original image to be scored under the corresponding current business type based on the corresponding R*P similarity results.

[0276] The state feature vector of the standard image is a state feature vector obtained by inputting the standard image and the label of the corresponding current business type into the state recognition model for feature extraction; the R*P standard images are typical images of unambiguous distinguishable state categories set for various states of the corresponding current business type, R is the total number of all state categories under the corresponding current business type, and P is a positive integer.

[0277] Optionally, in the state recognition unit, the state recognition model includes a first hierarchical network for semantic segmentation and a second hierarchical network for state recognition.

[0278] The input of the first hierarchical network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, which is used to mark the image regions related to each current business type.

[0279] The second hierarchical network includes M sub-networks corresponding to M business types one by one; wherein M is the total number of all business types.

[0280] The input of each sub-network is an original image to be scored and a modified semantic feature map corresponding to each label of the first current business type corresponding to the sub-network, each sub-network connects the original image to be scored and the modified semantic feature map in the channel dimension, and inputs the connected image into the classification network for state recognition to obtain a first state feature vector corresponding to the first current business type; wherein the modified semantic feature map is a semantic feature map up-sampled to the same size as the original image to be scored.

[0281] Optionally, to further accelerate the classification convergence and improve the robustness of the network, position information can be provided for the connected image in the state recognition unit. Specifically, the process of inputting the connected image into the classification network for state recognition can include:

[0282] dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each two-dimensional slice in each two-dimensional slice;

[0283] inputting all two-dimensional slices with added two-dimensional position information into the classification network for state recognition;

[0284] wherein, through training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is ensured to be greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

[0285] Optionally, to further improve the classification accuracy, an attention mechanism can be added to the classification network in the state recognition unit to assign a weight to each pixel input into the classification network, and the weight of the pixel located in the image region related to the current business type is greater than the weight of the pixel located outside the image region related to the current business type.

[0286] Optionally, Figure 6 The image quality scoring device shown can further include a training unit for training the state recognition model;

[0287] wherein, when the state recognition model is trained in the training unit, the training of the first level network can be completed using training sample images, and the second level network is trained based on the trained first level network; or,

[0288] When the state recognition model is trained in the training unit, the first level network and the second level network can be jointly trained; or,

[0289] In the training of the state recognition model by the training unit, the first hierarchical network is initially trained by using the training sample images, and the second hierarchical network is initially trained based on the first hierarchical network obtained after the initial training; after the initial training of the second hierarchical network, the first hierarchical network and the second hierarchical network are jointly trained.

[0290] In the joint training, the loss function is a weighted sum of the first loss function of the first hierarchical network and the second loss function of the second hierarchical network.

[0291] Optionally, in the joint training, the first loss function corresponding to each input service type can be calculated based on the loss weight of the input label; wherein the input service type is the service type to which the input label belongs.

[0292] In the calculation of the first loss function corresponding to any input service type, the loss weight of the input label of the any input service type is greater than the loss weight of the input label not belonging to the any input service type.

[0293] Optionally, in the state recognition unit, the process of determining the quality score result of the original image to be scored under the corresponding current service type can specifically include:

[0294] The weighted mean of the R*P similarity results is calculated, and the calculation result is taken as the quality score result of the original image to be scored under the corresponding current service type.

[0295] Corresponding to the second image quality scoring method, the present application provides an image quality scoring system which can be used to implement the second image quality scoring method, and the basic structure of the system is as shown in Figure 7 , and specifically includes an image quality scoring device and a training device, wherein the basic structures of the image quality scoring device and the training device in the image quality scoring system are respectively as shown in Figure 8 and Figure 9 .

[0296] Referring to Figure 7 , Figure 8 and Figure 9 , the training device includes a state recognition unit, a scoring unit and a first training unit.

[0297] The state recognition unit is configured to input the test sample images and the labels corresponding to each test service type into the pre-trained state recognition model, mark the image regions related to each test service type, and perform feature extraction based on the image regions to obtain the first state feature vector corresponding to each test service type.

[0298] The scoring unit is configured to calculate, for each first state feature vector, a similarity between the first state feature vector and a state feature vector of R*P standard images corresponding to the respective test service type, to obtain R*P similarity results; and to determine, for each first state feature vector, a quality score result of the test sample image under the respective test service type based on the R*P similarity results.

[0299] The first training unit is configured to train the quality score regression model by using the test sample image, the identifier of each current service type, and the quality score result of the test sample image under each current service type.

[0300] The state feature vector of the standard image is a state feature vector obtained by inputting the standard image and a label of the respective test service type into the state recognition model for feature extraction; the R*P standard images are typical images of state categories that can be unambiguously distinguished, which are set for various states of the respective test service type, R is a total number of all state categories under the respective test service type, and P is a positive integer.

[0301] The image quality scoring device comprises an input unit, a quality scoring unit, and an output unit.

[0302] The input unit is configured to receive an original image to be scored and an identifier of each current service type, and send the original image to be scored and the identifier of each current service type to the quality scoring unit.

[0303] The quality scoring unit is configured to input the received original image to be scored and the identifier of each current service type into the quality score regression model trained by the training device in advance, to obtain a quality score result of the original image to be scored under each current service type.

[0304] The output unit is configured to output the quality score result of the original image to be scored under each current service type.

[0305] Optionally, in the state recognition unit of the training device, the state recognition model can comprise a first hierarchical network for semantic segmentation and a second hierarchical network for state recognition.

[0306] The input of the first hierarchical network is the test sample image and the label, and the output is a semantic feature map corresponding to each label, which is used to mark an image region related to each test service type.

[0307] The second hierarchical network comprises M sub-networks corresponding to M service types one by one; M is a total number of all service types.

[0308] The input of each sub-network is a test sample image and a corrected semantic feature map corresponding to each label of a first test service type corresponding to the sub-network, each sub-network connects the test sample image and the corrected semantic feature map in a channel dimension, and inputs the connected image into a classification network for state recognition to obtain a first state feature vector corresponding to the first test service type; wherein the corrected semantic feature map is a semantic feature map up-sampled to the same size as the test sample image.

[0309] Optionally, in the state recognition unit of the training device, inputting the connected image into the classification network for state recognition comprises:

[0310] dividing the connected image into a plurality of two-dimensional slices (patches), and adding two-dimensional position information corresponding to each two-dimensional slice in each two-dimensional slice;

[0311] inputting all two-dimensional slices with added two-dimensional position information into the classification network for state recognition;

[0312] wherein, through training of the state recognition model, the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current service type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current service type.

[0313] Optionally, in the state recognition unit of the training device, an attention mechanism is added to the classification network to assign a weight to each pixel input into the classification network, and the weight of a pixel located in the image region related to the test service type is greater than the weight of a pixel located outside the image region related to the test service type.

[0314] Optionally, the training device can further comprise a second training unit for training the state recognition model.

[0315] wherein, when the second training unit trains the state recognition model, the training sample image is used to train the first hierarchical network, and the second hierarchical network is trained based on the trained first hierarchical network; or,

[0316] when the second training unit trains the state recognition model, the first hierarchical network and the second hierarchical network are jointly trained; or,

[0317] when the second training unit trains the state recognition model, the training sample image is used to initially train the first hierarchical network, the second hierarchical network is initially trained based on the first hierarchical network obtained after initial training, and the first hierarchical network and the second hierarchical network are jointly trained after initial training of the second hierarchical network.

[0318] In the joint training, the loss function is a weighted sum of the first loss function of the first hierarchical network and the second loss function of the second hierarchical network.

[0319] Optionally, in the joint training by the second training unit, the first loss function corresponding to each input service type is calculated based on the loss weight of all input labels; the input service type is a service type to which the input label belongs.

[0320] In the calculation of the first loss function corresponding to any input service type, the loss weight of the input label of the any input service type is greater than the loss weight of the input label not belonging to the any input service type.

[0321] Optionally, in the state recognition unit of the training device, the processing of determining the quality score result of the test sample image under the corresponding test service type can specifically include:

[0322] The weighted mean of the R*P similarity results is calculated, and the calculation result is taken as the quality score result of the test sample image under the corresponding test service type.

[0323] The application also provides a computer readable storage medium storing instructions, which, when executed by a processor, can perform the steps in the image quality scoring method as described above. In practical applications, the computer readable medium can be included in the devices / apparatuses / systems of the above embodiments, or can exist separately without being assembled into the devices / apparatuses / systems. The instructions stored in the computer readable storage medium can perform the steps in the image quality scoring method as described above when executed by the processor.

[0324] According to the embodiments disclosed in the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, but not for limiting the scope of protection of the application. In the embodiments disclosed in the application, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, apparatus or device.

[0325] Figure 10 The application further provides an electronic device. As shown in FIG. 8, a structural schematic diagram of an electronic device related to the embodiments of the application is shown, specifically: Figure 10

[0326] ​The electronic device can include a processor 1001 having one or more processing cores, a memory 1002 having one or more computer-readable storage media, and a computer program stored on the memory and executable on the processor. When the program of the memory 1002 is executed, the method of image quality scoring can be implemented.

[0327] Specifically, in actual application, the electronic device can further include a power supply 1003, an input and output unit 1004, and the like. Those skilled in the art can understand that, Figure 10 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them:

[0328] The processor 1001 is the control center of the electronic device, connects all parts of the electronic device through various interfaces and lines, executes various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 1002, and calls data stored in the memory 1002, thereby overall controlling the electronic device.

[0329] The memory 1002 can be used to store software programs and modules, i.e. the above-mentioned computer-readable storage medium. The processor 1001 executes various function applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function, etc.; the data storage area can store data created according to the use of the server, etc. In addition, the memory 1002 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 1002 can also include a memory controller to provide access for the processor 1001 to the memory 1002.

[0330] The electronic device further includes a power supply 1003 for supplying power to each component, which can be logically connected to the processor 1001 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 1003 can also include one or more direct or alternating current power supplies, a recharging system, a power supply failure detection circuit, a power supply converter or inverter, a power supply state indicator, and any other components.

[0331] The electronic device can further include an input / output unit 1004, which can be used to receive inputted digital or character information, and to generate a keyboard, mouse, joystick, optical signal input related to user settings and function control. The input / output unit 1004 can also be used to display information inputted by a user or provided to a user, and various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0332] The above description is merely that of the preferred embodiments of the application and is not to be taken in a limiting sense but is made merely for the purpose of providing some preferred embodiments of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the scope of the application.

Claims

1. An image quality scoring method, characterized in that, include: The original image to be scored and the label corresponding to each current business type are input into a pre-trained state recognition model. The image region related to each current business type is marked, and feature extraction is performed based on the image region to obtain a first state feature vector corresponding to each current business type. The state recognition model is trained with the label corresponding to the business type as supervision information. For each of the first state feature vectors, calculate the R value corresponding to the current service type. The similarity of the state feature vectors of P standard images is used to obtain R. P similarity results; For each of the first state feature vectors, for the R The weighted average of the P similarity results is calculated, and the result is used as the quality score of the original image to be scored under the corresponding current business type. The state feature vector of the standard image is: the state feature vector obtained by inputting the standard image and the corresponding label of the current business type into the state recognition model for feature extraction; R The P standard images are typical images that can unambiguously identify the state category for various state settings corresponding to the current business type, where R is the total number of all state categories under the current business type, and P is a positive integer. The state recognition model includes a first-level network for semantic segmentation and a second-level network for state recognition; The input to the first-level network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type; The second-level network includes M subnetworks, each corresponding to one of the M service types; where M is the total number of all service types. The input to each sub-network is the original image to be scored and the corrected semantic feature maps corresponding to each label of the first current service type corresponding to that sub-network. Each sub-network concatenates the original image to be scored and the corrected semantic feature maps along the channel dimension, and inputs the concatenated image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein, the corrected semantic feature map is a semantic feature map upsampled to the same size as the original image to be scored; The step of inputting the concatenated image into the classification network for state recognition includes: The concatenated image is divided into several two-dimensional slices (patch), and the two-dimensional position information corresponding to the two-dimensional slice is added to each two-dimensional slice. All 2D slices with added 2D location information are input into a classification network for state recognition. Specifically, through training the state recognition model, it is ensured that the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

2. The method according to claim 1, characterized in that, An attention mechanism is added to the classification network to assign weights to each pixel input to the classification network, and the weights of pixels located in the image region related to the current business type are greater than the weights of pixels located outside the image region related to the current business type.

3. The method according to claim 1, characterized in that, When training the state recognition model, the first-layer network is trained using training sample images, and the second-layer network is trained based on the trained first-layer network; or, When training the state recognition model, the first-level network and the second-level network are jointly trained; or, When training the state recognition model, the first-level network is initially trained using training sample images, and the second-level network is initially trained based on the first-level network obtained after the initial training. After initial training of the second-level network, the first-level network and the second-level network are jointly trained. During the joint training, the loss function is a weighted sum of the first loss function of the first layer network and the second loss function of the second layer network.

4. The method according to claim 3, characterized in that, During the joint training, the first loss function corresponding to each input service type is calculated based on the loss weights of all input labels; wherein, the input service type is the service type to which the input label belongs; When calculating the first loss function corresponding to any input service type, the loss weight of the input label of any input service type is greater than the loss weight of the input label that does not belong to any input service type.

5. The method according to claim 3 or 4, characterized in that, The first loss function is the Dice loss function, and the second loss function is the focal loss function.

6. The method according to claim 1, characterized in that, Each current service type and each state category includes P standard images, which include original images with different scene brightness, different shooting angles, and / or different postures under each state category.

7. An image quality scoring method, characterized in that, include: The original image to be scored and the identifier of each current business type are input into a pre-trained quality scoring regression model to obtain the quality score result of the original image to be scored under each current business type. The quality score regression model is a neural network model trained using test sample images, the identifier of each test service type, and the quality score results of the test sample images under each test service type. The process for determining the quality score of the test sample image under each test service type includes: The test sample images and labels corresponding to each test service type are input into a pre-trained state recognition model to label the image regions related to each test service type. Feature extraction is performed based on the image regions to obtain a first state feature vector corresponding to each test service type. The state recognition model is trained using the labels corresponding to the service type as supervision information. For each of the first state feature vectors, calculate the R value corresponding to the corresponding test service type. The similarity of the state feature vectors of P standard images is used to obtain R. P similarity results; For each of the first state feature vectors, for the R A weighted average is calculated from P similarity results, and the result is used as the quality score of the test sample image under the corresponding test service type. The state feature vector of the standard image is: the state feature vector obtained after feature extraction of the standard image and the corresponding test service type label into the state recognition model; R P standard images are typical images that can unambiguously identify state categories for various state settings corresponding to the corresponding test service type, where R is the total number of all state categories under the corresponding test service type, and P is a positive integer; The state recognition model includes a first-level network for semantic segmentation and a second-level network for state recognition; The input to the first-level network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type; The second-level network includes M subnetworks, each corresponding to one of the M service types; where M is the total number of all service types. The input to each sub-network is the original image to be scored and the corrected semantic feature maps corresponding to each label of the first current service type corresponding to that sub-network. Each sub-network concatenates the original image to be scored and the corrected semantic feature maps along the channel dimension, and inputs the concatenated image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein, the corrected semantic feature map is a semantic feature map upsampled to the same size as the original image to be scored; The step of inputting the concatenated image into the classification network for state recognition includes: The concatenated image is divided into several two-dimensional slices (patch), and the two-dimensional position information corresponding to the two-dimensional slice is added to each two-dimensional slice. All 2D slices with added 2D location information are input into a classification network for state recognition. Specifically, through training the state recognition model, it is ensured that the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

8. An image quality scoring device, characterized in that, include: Status recognition unit and scoring unit; The state recognition unit is used to input the original image to be scored and the label corresponding to each current business type into a pre-trained state recognition model, mark the image region related to each current business type, and extract features based on the image region to obtain a first state feature vector corresponding to each current business type; wherein, the state recognition model is trained with the label corresponding to the business type as supervision information; The scoring unit is used to calculate, for each of the first state feature vectors, the R-value corresponding to the first state feature vector and the corresponding test service type. The similarity of the state feature vectors of P standard images is used to obtain R. P similarity results; also used for each of the first state feature vectors, for the R The weighted average of the P similarity results is calculated, and the result is used as the quality score of the original image to be scored under the corresponding current business type. The state feature vector of the standard image is: the state feature vector obtained by inputting the standard image and the corresponding label of the current business type into the state recognition model for feature extraction; R The P standard images are typical images that can unambiguously identify the state category for various state settings corresponding to the current business type, where R is the total number of all state categories under the current business type, and P is a positive integer. In the state recognition unit, the state recognition model includes a first-level network for semantic segmentation and a second-level network for state recognition; The input to the first-level network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type; The second-level network includes M subnetworks, each corresponding to one of the M service types; where M is the total number of all service types. The input to each sub-network is the original image to be scored and the corrected semantic feature maps corresponding to each label of the first current service type corresponding to that sub-network. Each sub-network concatenates the original image to be scored and the corrected semantic feature maps along the channel dimension, and inputs the concatenated image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein, the corrected semantic feature map is a semantic feature map upsampled to the same size as the original image to be scored; In the state recognition unit, the step of inputting the concatenated image into the classification network for state recognition includes: The concatenated image is divided into several two-dimensional slices (patch), and the two-dimensional position information corresponding to the two-dimensional slice is added to each two-dimensional slice. All 2D slices with added 2D location information are input into a classification network for state recognition. Specifically, through training the state recognition model, it is ensured that the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

9. The apparatus according to claim 8, characterized in that, In the state recognition unit, an attention mechanism is added to the classification network to assign weights to each pixel input to the classification network, and the weights of pixels located in the image region related to the current service type are greater than the weights of pixels located outside the image region related to the current service type.

10. The apparatus according to claim 8, characterized in that, The device further includes a training unit for training and generating the state recognition model; Specifically, when the training unit trains the state recognition model, the first-layer network is trained using training sample images, and the second-layer network is trained based on the trained first-layer network; or... When the state recognition model is trained in the training unit, the first-level network and the second-level network are jointly trained; or, When the training unit trains the state recognition model, the first-level network is initially trained using training sample images, and the second-level network is initially trained based on the first-level network obtained after the initial training; after the second-level network is initially trained, the first-level network and the second-level network are jointly trained. During the joint training, the loss function is a weighted sum of the first loss function of the first layer network and the second loss function of the second layer network.

11. The apparatus according to claim 10, characterized in that, During the joint training, the first loss function corresponding to each input service type is calculated based on the loss weights of all input labels; wherein, the input service type is the service type to which the input label belongs; When calculating the first loss function corresponding to any input service type, the loss weight of the input label of any input service type is greater than the loss weight of the input label that does not belong to any input service type.

12. An image quality scoring system, characterized in that, Includes training equipment and image quality scoring equipment; The training device includes a state recognition unit, a scoring unit, and a first training unit; The state recognition unit is used to input the test sample image and the label corresponding to each test service type into the pre-trained state recognition model, mark the image region related to each test service type, and perform feature extraction based on the image region to obtain a first state feature vector corresponding to each test service type; wherein, the state recognition model is trained with the label corresponding to the service type as supervision information; The scoring unit is used to calculate, for each of the first state feature vectors, the R-value corresponding to the first state feature vector and the corresponding test service type. The similarity of the state feature vectors of P standard images is used to obtain R. P similarity results; also used for each of the first state feature vectors, for the R A weighted average is calculated from P similarity results, and the result is used as the quality score of the test sample image under the corresponding test service type. The first training unit is used to train and generate a quality score regression model using the test sample image, the identifier of each current business type, and the quality score result of the test sample image under each current business type. The image quality scoring device includes an input unit, a quality scoring unit, and an output unit; The input unit is used to receive the original image to be scored and the identifier of each current service type, and send them to the quality scoring unit; The quality scoring unit is used to input the received original image to be scored and the identifier of each current service type into the quality scoring regression model to obtain the quality score result of the original image to be scored under each current service type. The output unit is used to output the quality score result of the original image to be scored under each current service type; The state feature vector of the standard image is: the state feature vector obtained after feature extraction of the standard image and the corresponding test service type label into the state recognition model; R P standard images are typical images that can unambiguously identify state categories for various state settings corresponding to the corresponding test service type, where R is the total number of all state categories under the corresponding test service type, and P is a positive integer; In the state recognition unit, the state recognition model includes a first-level network for semantic segmentation and a second-level network for state recognition; The input to the first-level network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type; The second-level network includes M subnetworks, each corresponding to one of the M service types; where M is the total number of all service types. The input to each sub-network is the original image to be scored and the corrected semantic feature maps corresponding to each label of the first current service type corresponding to that sub-network. Each sub-network concatenates the original image to be scored and the corrected semantic feature maps along the channel dimension, and inputs the concatenated image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein, the corrected semantic feature map is a semantic feature map upsampled to the same size as the original image to be scored; In the state recognition unit, the step of inputting the concatenated image into the classification network for state recognition includes: The concatenated image is divided into several two-dimensional slices (patch), and the two-dimensional position information corresponding to the two-dimensional slice is added to each two-dimensional slice. All 2D slices with added 2D location information are input into a classification network for state recognition. Specifically, through training the state recognition model, it is ensured that the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

13. An image quality scoring device, characterized in that, include: Input unit, quality scoring unit, and output unit; The input unit is used to receive the original image to be scored and the identifier of each current service type, and send them to the quality scoring unit; The quality scoring unit is used to input the received original image to be scored and the identifier of each current service type into a quality scoring regression model pre-trained by a training device to obtain the quality score result of the original image to be scored under each current service type. The output unit is used to output the quality score result of the original image to be scored under each current service type; The training device includes a state recognition unit, a scoring unit, and a first training unit; The state recognition unit is used to input the test sample image and the label corresponding to each test service type into the pre-trained state recognition model, mark the image region related to each test service type, and perform feature extraction based on the image region to obtain a first state feature vector corresponding to each test service type; wherein, the state recognition model is trained with the label corresponding to the service type as supervision information; The scoring unit is used to calculate, for each of the first state feature vectors, the R-value corresponding to the first state feature vector and the corresponding test service type. The similarity of the state feature vectors of P standard images is used to obtain R. P similarity results; also used for each of the first state feature vectors, for the R A weighted average is calculated from P similarity results, and the result is used as the quality score of the test sample image under the corresponding test service type. The first training unit is used to train and generate the quality score regression model using the test sample image, the identifier of each current business type, and the quality score result of the test sample image under each current business type. The state feature vector of the standard image is: the state feature vector obtained after feature extraction of the standard image and the corresponding test service type label into the state recognition model; R P standard images are typical images that can unambiguously identify state categories for various state settings corresponding to the corresponding test service type, where R is the total number of all state categories under the corresponding test service type, and P is a positive integer; In the state recognition unit, the state recognition model includes a first-level network for semantic segmentation and a second-level network for state recognition; The input to the first-level network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type; The second-level network includes M subnetworks, each corresponding to one of the M service types; where M is the total number of all service types. The input to each sub-network is the original image to be scored and the corrected semantic feature maps corresponding to each label of the first current service type corresponding to that sub-network. Each sub-network concatenates the original image to be scored and the corrected semantic feature maps along the channel dimension, and inputs the concatenated image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein, the corrected semantic feature map is a semantic feature map upsampled to the same size as the original image to be scored; In the state recognition unit, the step of inputting the concatenated image into the classification network for state recognition includes: The concatenated image is divided into several two-dimensional slices (patch), and the two-dimensional position information corresponding to the two-dimensional slice is added to each two-dimensional slice. All 2D slices with added 2D location information are input into a classification network for state recognition. Specifically, through training the state recognition model, it is ensured that the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

14. A training device, characterized in that, include: State recognition unit, scoring unit, and first training unit; The state recognition unit is used to input the test sample image and the label corresponding to each test service type into the pre-trained state recognition model, mark the image region related to each test service type, and perform feature extraction based on the image region to obtain a first state feature vector corresponding to each test service type; wherein, the state recognition model is trained with the label corresponding to the service type as supervision information; The scoring unit is used to calculate, for each of the first state feature vectors, the R-value corresponding to the first state feature vector and the corresponding test service type. The similarity of the state feature vectors of P standard images is used to obtain R. P similarity results; also used for each of the first state feature vectors, for the R A weighted average is calculated from P similarity results, and the result is used as the quality score of the test sample image under the corresponding test service type. The first training unit is used to train and generate a quality score regression model using the test sample image, the identifier of each current business type, and the quality score result of the test sample image under each current business type. The quality scoring model processes the input original image to be scored and the identifier of each current service type to obtain the quality score of the original image to be scored under each current service type; the state feature vector of the standard image is: the state feature vector obtained after feature extraction of the standard image and the label of the corresponding test service type by inputting them into the state recognition model; R P standard images are typical images that can unambiguously identify state categories for various state settings corresponding to the corresponding test service type, where R is the total number of all state categories under the corresponding test service type, and P is a positive integer; In the state recognition unit, the state recognition model includes a first-level network for semantic segmentation and a second-level network for state recognition; The input to the first-level network is the original image to be scored and the label, and the output is a semantic feature map corresponding to each label, used to mark the image region related to each current business type; The second-level network includes M subnetworks, each corresponding to one of the M service types; where M is the total number of all service types. The input to each sub-network is the original image to be scored and the corrected semantic feature maps corresponding to each label of the first current service type corresponding to that sub-network. Each sub-network concatenates the original image to be scored and the corrected semantic feature maps along the channel dimension, and inputs the concatenated image into a classification network for state recognition to obtain the first state feature vector corresponding to the first current service type; wherein, the corrected semantic feature map is a semantic feature map upsampled to the same size as the original image to be scored; In the state recognition unit, the step of inputting the concatenated image into the classification network for state recognition includes: The concatenated image is divided into several two-dimensional slices (patch), and the two-dimensional position information corresponding to the two-dimensional slice is added to each two-dimensional slice. All 2D slices with added 2D location information are input into a classification network for state recognition. Specifically, through training the state recognition model, it is ensured that the weight of the two-dimensional position information corresponding to the two-dimensional slice located in the image region related to the current business type is greater than the weight of the two-dimensional position information corresponding to the two-dimensional slice located outside the image region related to the current business type.

15. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed by the processor, it can implement the image quality scoring method according to any one of claims 1 to 7.

16. An electronic device, characterized in that, The electronic device includes at least a computer-readable storage medium and a processor; The processor is configured to read the executable instructions from the computer-readable storage medium and execute the instructions to implement the image quality scoring method according to any one of claims 1 to 7.

Images