Image processing method, device and storage medium

Abstract

The present invention discloses an image processing method, device and storage medium. The method includes: acquiring a first image and a second image; determining a detection image based on the first image and the second image; using at least one image evaluation model to identify the detection image to obtain at least one recognition result; each recognition result includes a score representing the quality of the detection image.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the advent of the fifth-generation mobile communication technology (5G) and the development of mobile Internet, image-related applications remain core applications. Therefore, how to evaluate image quality is a fundamental issue.

[0003] Among the existing image quality evaluation annotation technologies, either the scores are given directly by human eyes, which is time-consuming and labor-intensive, or the essential problem of image quality loss is not excavated, and the quality loss caused by image processing cannot be clearly identified. Summary of the Invention

[0004] In view of this, the main object of the present invention is to provide an image processing method, device and storage medium.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides an image processing method, the method comprising:

[0007] Acquire a first image and a second image;

[0008] determining a detection image according to the first image and the second image;

[0009] At least one image evaluation model is used to identify the detection image to obtain at least one recognition result; each of the recognition results includes a score representing the quality of the detection image.

[0010] In the above solution, the method further includes: generating an image evaluation model; the generating of the image evaluation model includes:

[0011] Obtain at least one training sample; each of the training samples is obtained by subtracting a first training image from a second training image; each of the training samples has a corresponding label;

[0012] Obtaining a training sample set according to the at least one training sample;

[0013] A preset neural network is obtained, and the preset neural network is trained using the training sample set and the label corresponding to each training sample to obtain an image evaluation model.

[0014] In the above solution, determining the detection image based on the first image and the second image includes:

[0015] Processing the first image using a preset image processing method to obtain a first target image;

[0016] Processing the second image using a preset image processing method to obtain a second target image;

[0017] Subtracting the first target image from the second target image to obtain the detection image.

[0018] In the above solution, when the image evaluation model is a fuzzy evaluation model, determining the detection image based on the first image and the second image includes: using an oblique filter to extract an edge information map of the first image as a first target image;

[0019] Using an oblique filter, extracting an edge information map of the second image as a second target image;

[0020] The first target image is subtracted from the second target image to obtain an edge information loss map as the detection image.

[0021] In the above solution, when the image evaluation model is a blockiness evaluation model, determining the detection image based on the first image and the second image includes:

[0022] Using horizontal and vertical filters, extracting an edge information map of the first image as a first target image;

[0023] Applying horizontal and vertical filters to extract an edge information map of the second image as a second target image;

[0024] The first target image is subtracted from the second target image to obtain an edge information enhanced image as the detection image.

[0025] In the above solution, the method further includes:

[0026] Performing absolute value processing on the detection image;

[0027] Accordingly, using at least one image evaluation model to identify the detection image includes:

[0028] At least one image evaluation model is used to identify the detection image after absolute value processing.

[0029] In the above solution, the method further includes:

[0030] When it is determined that there is a first recognition result that does not meet the preset condition in the at least one recognition result, determining a cause of the image quality problem according to the first recognition result;

[0031] An adjustment plan is determined based on the determined cause of the image quality problem; the adjustment plan is used to adjust a plan for obtaining the second image by processing the first image.

[0032] An embodiment of the present invention provides an image processing device, comprising:

[0033] A first processing module, configured to acquire a first image and a second image;

[0034] a second processing module, configured to determine a detection image based on the first image and the second image;

[0035] The third processing module is used to use at least one image evaluation model to identify the detection image and obtain at least one recognition result; each of the recognition results includes a score that characterizes the quality of the detection image.

[0036] In the above solution, the device further comprises: a pre-processing module for generating an image evaluation model;

[0037] The preprocessing module is specifically used to obtain at least one training sample; each training sample is obtained by subtracting a first training image from a second training image; each training sample has a corresponding label;

[0038] Obtaining a training sample set according to the at least one training sample;

[0039] A preset neural network is obtained, and the preset neural network is trained using the training sample set and the label corresponding to each training sample to obtain an image evaluation model.

[0040] In the above solution, the second processing module is used to process the first image using a preset image processing method to obtain a first target image;

[0041] Processing the second image using a preset image processing method to obtain a second target image;

[0042] Subtracting the first target image from the second target image to obtain the detection image.

[0043] In the above solution, when the image evaluation model is a fuzzy evaluation model, the second processing module is specifically configured to use an oblique filter to extract an edge information map of the first image as the first target image;

[0044] Using an oblique filter, extracting an edge information map of the second image as a second target image;

[0045] The first target image is subtracted from the second target image to obtain an edge information loss map as the detection image.

[0046] In the above solution, when the image evaluation model is a blockiness evaluation model, the second processing module is specifically configured to use a horizontal and vertical filter to extract an edge information map of the first image as the first target image;

[0047] Applying horizontal and vertical filters to extract an edge information map of the second image as a second target image;

[0048] The first target image is subtracted from the second target image to obtain an edge information enhanced image as the detection image.

[0049] In the above solution, the second processing module is further used to perform absolute value processing on the detection image;

[0050] Correspondingly, the third processing module is used to use at least one image evaluation model to identify the detection image after absolute value processing.

[0051] In the above solution, the apparatus further includes: a fourth processing module, configured to, when determining that there is a first recognition result that does not meet a preset condition in the at least one recognition result, determine a cause of the image quality problem according to the first recognition result;

[0052] An adjustment plan is determined based on the determined cause of the image quality problem; the adjustment plan is used to adjust a plan for obtaining the second image by processing the first image.

[0053] An embodiment of the present invention provides an image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-described image processing methods when executing the program.

[0054] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any one of the above image processing methods when executed by a processor.

[0055] An embodiment of the present invention provides an image processing method, device and storage medium, the method comprising: acquiring a first image and a second image; determining a detection image based on the first image and the second image; identifying the detection image using at least one image evaluation model to obtain at least one recognition result; each recognition result includes a score representing the quality of the detection image; in this way, image quality is intelligently and objectively evaluated on the basis of improving evaluation efficiency and reducing evaluation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic flow chart of an image processing method provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of a training method for an image evaluation model provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a method for obtaining a training sample set provided by an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of a training process provided by an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of a scoring method provided by an embodiment of the present invention;

[0061] Figure 6 A schematic diagram of an image processing system provided by an embodiment of the present invention;

[0062] Figure 7 A schematic diagram of another image processing system provided by an embodiment of the present invention;

[0063] Figure 8 A schematic structural diagram of an image processing device provided by an embodiment of the present invention;

[0064] Figure 9 A schematic structural diagram of another image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described in detail with reference to the embodiments, and the related technologies will be described first.

[0066] Image quality assessment can be divided into subjective and objective evaluation. The most traditional and reliable method for image quality evaluation is subjective evaluation, in which multiple experts provide scores and take the average to obtain the image quality score, also known as the Mean Opinion Score (MOS). However, subjective evaluation requires a significant amount of manpower and resources, making it difficult to apply in today's big data environment. Therefore, machine scoring methods, or objective evaluation of image quality, are becoming a major trend. Currently, the most commonly used objective evaluation method is based on the Peak Signal-to-Noise Ratio (PSNR). This method takes the difference between two images and divides it by the peak value. While this method is simple to implement, it differs significantly from the subjective perception of the human eye.

[0067] The objective evaluation of image quality is divided into full-reference, semi-reference and no-reference methods.

[0068] In the prior art, there is a method for intelligent image labeling that uses a deep learning algorithm for intelligent detection. The deep learning intelligent detection algorithm is called for intelligent labeling to obtain the category, position, and confidence of the labeled object in the image. This method does not reflect the essence of image quality and relies on the intelligent detection algorithm, which is unreliable.

[0069] A method for image quality scoring is also provided, which specifically provides a method for labeling any image in the data set as a quality comparison image. This method is not reliable because the image quality of any image needs to be determined first before other images can be compared. This method is aimed at evaluating the quality of images without a reference.

[0070] The aforementioned annotation techniques either rely on direct human scoring, which is time-consuming and labor-intensive, or fail to identify the underlying causes of image quality loss and clearly define the quality loss caused by image processing. Furthermore, the evaluation metrics don't directly correspond to human vision. For example, sub-items such as clarity and blocking artifacts, as perceived by the human eye, are not clearly addressed. This lacks intuitiveness and fails to truly improve annotation efficiency or evaluation accuracy.

[0071] Based on this, the method provided in an embodiment of the present invention obtains a first image and a second image; determines a detection image based on the first image and the second image; uses at least one image evaluation model to identify the detection image to obtain at least one recognition result; each of the recognition results includes a score that characterizes the quality of the detection image.

[0072] The present invention will be further described in detail below with reference to the embodiments.

[0073] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention; Figure 1 As shown, the method can be applied to a smart device, such as a server, and the method includes:

[0074] Step 101: Acquire a first image and a second image;

[0075] Step 102: Determine a detection image based on the first image and the second image;

[0076] Step 103: Use at least one image evaluation model to identify the detection image to obtain at least one recognition result; each recognition result includes a score representing the quality of the detection image.

[0077] Specifically, the first image is an original image, and the second image is an image to be evaluated obtained by processing the first image based on certain reasons (such as image encoding, image transmission, etc.).

[0078] By determining the oblique edge information loss (for blurring issues) or horizontal and vertical edge information enhancement (for blocking issues) of the image quality from the first image to the second image, the oblique edge information loss part and the horizontal and vertical edge information enhancement part are judged to determine the quality of the second image.

[0079] In one embodiment, determining a detection image based on the first image and the second image includes:

[0080] Processing the first image using a preset image processing method to obtain a first target image;

[0081] Processing the second image using a preset image processing method to obtain a second target image;

[0082] Subtracting the first target image from the second target image to obtain the detection image.

[0083] The subtraction of the two images is to perform a subtraction operation on corresponding pixels between the two images.

[0084] The evaluation of image quality includes at least one of the following components:

[0085] Blurring, such as blurred edges (i.e., loss of detail);

[0086] Blocking artifacts, also known as mosaics, can be caused by missing key information or by encoding in macroblocks during the encoding and decoding process. Specifically, block-based transform coding is widely used in image compression. As the bit rate decreases, quantization becomes coarser, resulting in discontinuities at block boundaries, creating noticeable defects in the reconstructed image. This is known as blocking artifacts.

[0087] Accordingly, the at least one image evaluation model may include: a blur evaluation model, a block effect evaluation model, etc.

[0088] Specifically, for the evaluation of the fuzzy component, the image evaluation model includes a fuzzy evaluation model;

[0089] The preset image processing method uses an oblique filter for edge detection; for example, the oblique filter is obtained by improving the Sobel operator.

[0090] For example, the Sobel operator for the horizontal and vertical filters can be:

[0091] The sobel operator of the oblique filter can be:

[0092] Specifically, when the image evaluation model is a fuzzy evaluation model, determining the detection image based on the first image and the second image includes:

[0093] Using an oblique filter, extracting an edge information map of the first image as a first target image;

[0094] Using an oblique filter, extracting an edge information map of the second image as a second target image;

[0095] The first target image is subtracted from the second target image to obtain an edge information loss map as the detection image.

[0096] Here, the edge information map extracted using the oblique filter can represent the oblique edge information portion;

[0097] The obtained edge information loss map represents the oblique edge information loss portion corresponding to the image.

[0098] The pixels of the edge information map of the original image are subtracted from the pixels corresponding to the edge information map of the image to be evaluated, and the resulting detection image is the edge information loss map.

[0099] Specifically, for the evaluation of blockiness classification, the image evaluation model includes a blockiness evaluation model;

[0100] The preset image processing method adopts a blocking effect detection method using horizontal and vertical filters;

[0101] Specifically, when the image evaluation model is a blockiness evaluation model, determining the detection image according to the first image and the second image includes:

[0102] Using horizontal and vertical filters, extracting an edge information map of the first image as a first target image;

[0103] Applying horizontal and vertical filters to extract an edge information map of the second image as a second target image;

[0104] The first target image is subtracted from the second target image to obtain an edge information enhanced image as the detection image.

[0105] Here, the edge information map extracted by using horizontal and vertical filters can represent the horizontal and vertical edge information parts;

[0106] The obtained edge information enhancement map represents the horizontal and vertical edge information enhancement parts corresponding to the image.

[0107] That is, the pixels of the image to be evaluated are subtracted from the corresponding pixels of the original image; the resulting detection image is the edge information enhanced image.

[0108] A detection image is obtained by subtracting the two images. The detection image reflects the difference information between the two images. The difference information is evaluated to reflect the image quality of the second image.

[0109] Here, considering the evaluation for different problems, different filters are used for image processing, and the detection image obtained by subtracting the processed images can better reflect the problem.

[0110] In some embodiments, the method further comprises:

[0111] Performing absolute value processing on the detection image;

[0112] Using at least one image evaluation model to identify the detection image includes:

[0113] At least one image evaluation model is used to identify the detection image after absolute value processing.

[0114] In one embodiment, the method further comprises:

[0115] When it is determined that there is a first recognition result that does not meet the preset condition in the at least one recognition result, determining a cause of the image quality problem according to the first recognition result;

[0116] Based on the determined cause of the image quality problem, an adjustment plan is determined; the adjustment plan is used to adjust the plan for obtaining the second image by processing the first image. In order to achieve image evaluation, a method for image quality evaluation using an image evaluation model is proposed.

[0117] Specifically, considering that in actual applications, images will be processed during image transmission and encoding and decoding, for example, a first image is compressed to obtain a second image and the second image is transmitted, and the first image is encoded and decoded to obtain a second image; after image quality evaluation is performed using the method provided in an embodiment of the present invention, it can be determined whether there is a problem with the obtained second image. When it is determined that a problem occurs, the schemes adopted in the encoding and decoding process and the image compression process can be adjusted to obtain a second image that meets the corresponding requirements.

[0118] The recognition result includes: at least one score and a confidence level corresponding to each score.

[0119] The determining that there is a first recognition result that does not meet a preset condition in the at least one recognition result includes:

[0120] comparing a first score with the highest confidence level in a recognition result obtained by using the fuzzy evaluation model to recognize the detection image with a preset first threshold value, and determining that the recognition result obtained by using the fuzzy evaluation model to recognize the detection image does not meet a preset condition when the first score with the highest confidence level is determined to be lower than the preset first threshold value;

[0121] Comparing a second score with the highest confidence in a recognition result obtained by using the blockiness evaluation model to recognize the detection image with a preset second threshold, and determining that the second score with the highest confidence is lower than the preset second threshold, determining that the recognition result obtained by using the blockiness evaluation model to recognize the detection image does not meet the preset condition.

[0122] In this way, corresponding adjustment solutions can be determined for the problems of blur components and block effect components.

[0123] The method provided in the embodiment of the present invention is aimed at full-reference image quality evaluation, that is, comparing the image to be evaluated with the original image to obtain an objective score, which can be used to monitor systems such as image encoding and decoding and transmission and storage, and use this evaluation result as a standard to adaptively adjust system parameters such as image encoding and decoding and transmission and storage.

[0124] In one embodiment, the method further includes: generating an image evaluation model; the generating the image evaluation model includes:

[0125] Obtain at least one training sample; each of the training samples is obtained by subtracting a first training image from a second training image; each of the training samples has a corresponding label;

[0126] Obtaining a training sample set according to the at least one training sample;

[0127] A preset neural network is obtained, and the preset neural network is trained using the training sample set and the label corresponding to each training sample to obtain an image evaluation model.

[0128] Here, considering that the main problems caused by image encoding and transmission are loss of details (i.e., edge blur), blocking effects (i.e., mosaics) caused by loss of key information, or blocking effects caused by encoding in macroblock units during the encoding and decoding process, at least a blur evaluation model and a blocking effect evaluation model are provided to detect edge blur.

[0129] The labels are scored by developers or trainers based on corresponding rules and experience.

[0130] The training steps of the two models are the same. The difference is that for the fuzzy component, that is, for the fuzzy evaluation model, the training samples are obtained by subtracting the first training image from the second training image, including:

[0131] Using an oblique filter, extract the edge information map of the sample original image as the first training image;

[0132] Using an oblique filter, extract the edge information map of the sample image to be evaluated as the second training image;

[0133] The first training image is subtracted from the second training image to obtain a sample edge information loss map as the training sample.

[0134] The corresponding labels are the scores for the fuzzy questions.

[0135] For the blocking effect component, that is, the blocking effect evaluation model, a training sample is obtained by subtracting the first training image from the second training image, including:

[0136] Use horizontal and vertical filters to extract the edge information map of the sample original image as the first training image;

[0137] Use horizontal and vertical filters to extract the edge information map of the sample image to be evaluated as the second training image;

[0138] The first training image is subtracted from the second training image to obtain a sample edge information enhanced image as the training sample.

[0139] The corresponding labels are the scores for the blocking effect problem.

[0140] The training process may further include: taking absolute value processing of the training samples;

[0141] Correspondingly, during the training process, the training image after absolute value processing is used to perform model training.

[0142] The following uses the fuzzy evaluation model as an example to illustrate a specific training method for an image evaluation model.

[0143] Figure 2 A schematic diagram of a training method for an image evaluation model provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0144] Step 201: Acquire at least one training sample, and obtain a training sample set based on the at least one training sample;

[0145] Step 202: Obtain a preset neural network, and use the training sample set and the label corresponding to each training sample to train the preset neural network to obtain an image evaluation model.

[0146] Specifically, step 201 includes:

[0147] Step 2011: Process the sample original image and the sample image to be evaluated with an oblique filter to extract edge information, thereby obtaining a sample edge information map of the sample original image and a sample edge information map of the sample image to be evaluated;

[0148] The oblique filter is obtained by improving the Sobel operator of the horizontal and vertical filters.

[0149] For example, the Sobel operator for the horizontal and vertical filters can be:

[0150] The sobel operator of the improved oblique filter can be:

[0151] Here, by obtaining a sample original image and a sample image to be evaluated, the sample original image and the sample image to be evaluated are subjected to the same filtering process. Here, based on the measurement requirements of each component, such as blur and blocking effect, it is not limited to these two components. Corresponding components that can use filters to extract corresponding information can be adopted in the following manner. If the edge detection required for blur measurement is extracted, the image processing method corresponding to blur evaluation is used (i.e., extraction is performed using an oblique filter). If the amount of blocking effect is evaluated, the image processing method for blocking effect is used (i.e., extraction is performed using a horizontal and vertical filter).

[0152] Step 2012: Subtract the sample edge information map of the sample original image from the sample edge information map of the sample image to be evaluated to obtain a sample edge information loss map;

[0153] Here, the sample edge information loss map represents the oblique edge information loss portion of the sample.

[0154] Step 2013: The labeler uses the sample edge information loss map as a reference and re-examines the sample original image and the sample to be evaluated image to obtain a quality evaluation score as a label.

[0155] Here, we obtain the sample edge extraction map after processing of the original sample image and the sample image to be evaluated, and compare the difference between the two to obtain the visualization of the indicators of each evaluation component. The annotation content is the difference between the edge map extracted for the evaluation component of the original sample image and the sample image to be evaluated, which is convenient for the annotation personnel to perform annotation and image quality evaluation model training.

[0156] Step 2014: A set of sample edge information loss maps and corresponding quality evaluation scores (ie, labels) are taken as a training sample and added to the training sample set.

[0157] The operations of steps 2011 to 2014 are performed on different groups of sample original images and sample images to be evaluated to obtain a training sample set including a plurality of training samples.

[0158] Specifically, step 202 includes:

[0159] Step 2021: Obtain a fuzzy component data set (ie, the above-mentioned training sample set); wherein, the image included in each training sample is a sample edge information loss map, and the label is a quality evaluation score.

[0160] Step 2022: To ensure that the image quality is not lost, the image resolution is not adjusted and is directly input to the Resnet101 network;

[0161] Here, the model is not limited to the Resnet101 network, and any classification model that does not restrict the input size can be used to obtain basic texture information. Taking the Resnet101 network as an example, step 2022 outputs 1x1x2048 data.

[0162] Step 2023: The Flatten module reduces the 1x1x2048 feature map to a 1-dimensional array of 2048 values ​​as the input of the full connection.

[0163] Step 2024: The fully connected layer is used to filter the feature information required for fuzzy evaluation;

[0164] The fully connected layer input is 2048 input units and the output is n output units. The value of n is the same as the number of quality evaluation scores. The following description uses 5 quality evaluation scores for illustration.

[0165] Step 2025: Use the softmax classifier to obtain the fuzzy score.

[0166] You can set the value range as needed, with the score being a positive integer. For example, 1-5, where 1 represents high distortion and 5 represents low distortion. However, this range is not limited to 1-5. You can increase or decrease the score discrimination based on your specific needs.

[0167] Step 2026: The loss function uses the cross entropy function corresponding to softmax and uses the gradient descent training method.

[0168] Step 2027: Save the trained fuzzy evaluation model.

[0169] The above example is used to illustrate the training method of the model evaluation model. The above method can also be used to train the block effect evaluation model. The difference is: extract the edge information enhancement map (representing the horizontal and vertical edge information enhancement part, using horizontal and vertical filters for extraction), and replace the blur annotation with the block effect annotation (that is, the annotation staff’s scoring standards can be different). The result after training is the block effect evaluation model.

[0170] In an embodiment of the present invention, the evaluation models corresponding to the above-mentioned at least two quality indicators can be trained in sequence, but are not limited to the evaluation models of the above-mentioned two quality indicators. The quality loss generated during the image processing process can be trained using the above-mentioned method to obtain a corresponding quality loss evaluation model, which will not be repeated here.

[0171] Figure 3 A schematic diagram of a method for obtaining a training sample set provided by an embodiment of the present invention; Figure 3 As shown, the method includes: obtaining a training sample set; the training sample set includes: at least one training sample; the training sample is a sample edge information loss map; scoring each sample edge information loss map as a label.

[0172] For each training sample, the sample original image is filtered to obtain the first sample edge information map of the sample original image; the sample image to be evaluated is filtered to obtain the second sample edge information map of the sample to be evaluated; the first sample edge information map and the second sample edge information map are subtracted and normalized to obtain the sample edge information loss map.

[0173] Taking the fuzzy component as an example, the training process is further explained. Figure 4 As shown, specifically including:

[0174] A fuzzy component evaluation dataset is obtained; an image in the fuzzy component evaluation dataset is an edge information loss map; and a label is added to the edge information loss map, wherein the label is a quality evaluation score.

[0175] To ensure image quality, the image resolution is not adjusted and is directly input to the ResNet101 network (this is not limited to ResNet101 networks; any classification model with no input size restrictions can be used to obtain basic texture information. For example, the output of the ResNet101 network is 1x1x2048 data). The Flatten module reduces the 1x1x2048 feature map to a 1-dimensional array of 2048 values, which serves as the input to the fully connected layer (the fully connected layer is used to filter the feature information required for fuzzy evaluation; the fully connected layer input is 2048 input units and outputs n output units, where n is the same as the quality score). A softmax classifier is used to obtain the fuzzy score. The loss function uses the cross-entropy function corresponding to softmax, and the gradient descent algorithm based on cross-entropy is used for training to obtain the fuzzy evaluation model.

[0176] Figure 5 A schematic diagram of a scoring method provided by an embodiment of the present invention; Figure 5 As shown, the method includes:

[0177] Performing filtering on the original image to obtain a first edge information map of the original image;

[0178] Performing filtering on the image to be evaluated to obtain a second edge information map of the image to be evaluated;

[0179] Subtracting the first edge information map from the second edge information map and performing normalization processing to obtain an edge information loss map;

[0180] The edge information loss graph is inferred using a fuzzy evaluation model to obtain an evaluation score.

[0181] Figure 6 A schematic diagram of an image processing system provided by an embodiment of the present invention; Figure 6 As shown, the system includes a transmission device, which is used to transmit the original video. During the transmission process, the original video needs to be processed to obtain the video to be evaluated;

[0182] Using an image processing device to process images of corresponding frames in the original video and the video to be evaluated; the image processing device uses the image processing method provided by an embodiment of the present invention;

[0183] By processing the corresponding frame image, it is determined whether the link, transmission equipment, network parameters, etc. of the transmission device need to be adjusted.

[0184] Here, considering that packet loss may occur in the network transmission device, the impact on video quality is different depending on whether the lost packet contains key information in the video packet. The image processing method provided in the embodiment of the present invention can be used to evaluate the video quality and make adjustments based on the evaluation results.

[0185] Figure 7 A schematic diagram of another image processing system provided by an embodiment of the present invention; Figure 7 As shown, the system includes a compression encoding device, which is used to compress and encode the original image to obtain an image to be evaluated;

[0186] Using an image processing device to process the original image and the image to be evaluated; the image processing device uses the image processing method provided by an embodiment of the present invention;

[0187] By processing the corresponding frame image, it is determined whether the compression encoding device needs to be adjusted.

[0188] Taking the h264 encoder as an example, if the blur value does not meet the requirements, the quantization parameter (QP) value can be adjusted; if the blocking effect is obvious, you can consider improving modules such as the loop filter.

[0189] Figure 8 A schematic structural diagram of an image processing device provided by an embodiment of the present invention; Figure 8 As shown, the device includes:

[0190] A first processing module, configured to acquire a first image and a second image;

[0191] a second processing module, configured to determine a detection image based on the first image and the second image;

[0192] The third processing module is used to use at least one image evaluation model to identify the detection image and obtain at least one recognition result; each of the recognition results includes a score that characterizes the quality of the detection image.

[0193] In some embodiments, the apparatus further comprises: a pre-processing module for generating an image evaluation model;

[0194] The preprocessing module is specifically used to obtain at least one training sample; each training sample is obtained by subtracting a first training image from a second training image; each training sample has a corresponding label;

[0195] Obtaining a training sample set according to the at least one training sample;

[0196] A preset neural network is obtained, and the preset neural network is trained using the training sample set and the label corresponding to each training sample to obtain an image evaluation model.

[0197] In some embodiments, the second processing module is configured to process the first image using a preset image processing method to obtain a first target image;

[0198] Processing the second image using a preset image processing method to obtain a second target image;

[0199] Subtracting the first target image from the second target image to obtain the detection image.

[0200] In some embodiments, corresponding to when the image evaluation model is a fuzzy evaluation model, the second processing module is specifically configured to use an oblique filter to extract an edge information map of the first image as the first target image;

[0201] Using an oblique filter, extracting an edge information map of the second image as a second target image;

[0202] The first target image is subtracted from the second target image to obtain an edge information loss map as the detection image.

[0203] In some embodiments, when the image evaluation model is a blockiness evaluation model, the second processing module is specifically configured to use a horizontal and vertical filter to extract an edge information map of the first image as the first target image;

[0204] Applying horizontal and vertical filters to extract an edge information map of the second image as a second target image;

[0205] The first target image is subtracted from the second target image to obtain an edge information enhanced image as the detection image.

[0206] In some embodiments, the second processing module is further configured to perform absolute value processing on the detection image;

[0207] Correspondingly, the third processing module is used to use at least one image evaluation model to identify the detection image after absolute value processing.

[0208] In some embodiments, the apparatus further comprises: a fourth processing module configured to, when determining that there is a first recognition result that does not meet a preset condition in the at least one recognition result, determine a cause of the image quality problem according to the first recognition result;

[0209] An adjustment plan is determined based on the determined cause of the image quality problem; the adjustment plan is used to adjust a plan for obtaining the second image by processing the first image.

[0210] It should be noted that the image processing devices provided in the above embodiments, when implementing the corresponding image processing methods, are illustrated by the division of the aforementioned program modules. In actual applications, the aforementioned processing can be assigned to different program modules as needed, that is, the internal structure of the corresponding device can be divided into different program modules to complete all or part of the aforementioned processing. Furthermore, the devices provided in the above embodiments and the corresponding method embodiments are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0211] Figure 9 A structural diagram of another image processing device provided by an embodiment of the present invention is shown in FIG. Figure 9 As shown, the device 90 includes: a processor 901 and a memory 902 for storing a computer program that can be run on the processor; when the processor 901 is used to run the computer program, it executes: acquiring a first image and a second image; determining a detection image based on the first image and the second image; using at least one image evaluation model to identify the detection image to obtain at least one recognition result; each of the recognition results includes a score that characterizes the quality of the detection image.

[0212] In one embodiment, the processor 901 is further configured to, when running the computer program, execute the following: obtaining at least one training sample; each training sample is obtained by subtracting a first training image from a second training image; each training sample corresponds to a label; obtaining a training sample set based on the at least one training sample; obtaining a preset neural network, and using the training sample set and the label corresponding to each training sample to train the preset neural network to obtain an image evaluation model.

[0213] In one embodiment, the processor 901 is further used to execute, when running the computer program: using a preset image processing method to process the first image to obtain a first target image; using a preset image processing method to process the second image to obtain a second target image; subtracting the first target image from the second target image to obtain the detection image.

[0214] In one embodiment, the processor 901 is further configured to, when running the computer program, execute the following steps: extracting an edge information map of the first image using an oblique filter as a first target image; extracting an edge information map of the second image using an oblique filter as a second target image; and subtracting the second target image from the first target image to obtain an edge information loss map as the detection image.

[0215] In one embodiment, the processor 901 is further used to execute, when running the computer program: using horizontal and vertical filters to extract the edge information map of the first image as the first target image; using horizontal and vertical filters to extract the edge information map of the second image as the second target image; subtracting the first target image from the second target image to obtain an edge information enhanced map as the detection image.

[0216] In one embodiment, the processor 901 is further configured to, when running the computer program, execute: performing absolute value processing on the detection image; and correspondingly, using at least one image evaluation model to identify the detection image after the absolute value processing.

[0217] In one embodiment, the processor 901 is also used to execute the following when running the computer program: when it is determined that there is a first recognition result that does not meet the preset conditions in the at least one recognition result, determining the cause of the image quality problem based on the first recognition result; determining an adjustment plan based on the determined cause of the image quality problem; the adjustment plan is used to adjust the plan for obtaining the second image by processing the first image.

[0218] In actual application, the device 90 may further include: at least one network interface 903. The various components in the device 90 are coupled together via a bus system 904. It is understood that the bus system 904 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 9 In the figure, various buses are labeled as bus system 904. There may be at least one processor 901. The network interface 903 is used for wired or wireless communication between the apparatus 90 and other devices.

[0219] The memory 902 in this embodiment of the present invention is used to store various types of data to support the operation of the device 90 .

[0220] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 901. Processor 901 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 901 or by software instructions. The above processor 901 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. Processor 901 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium located in memory 902. Processor 901 reads information from memory 902 and, in conjunction with its hardware, completes the steps of the above method.

[0221] In an exemplary embodiment, the device 90 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0222] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon; when the computer program is executed by a processor, the computer program performs the following steps: acquiring a first image and a second image; determining a detection image based on the first image and the second image; identifying the detection image using at least one image evaluation model to obtain at least one recognition result; each recognition result includes a score representing the quality of the detection image.

[0223] In one embodiment, when the computer program is executed by a processor, the following steps are performed: obtaining at least one training sample; each training sample is obtained by subtracting a first training image from a second training image; each training sample corresponds to a label; obtaining a training sample set based on the at least one training sample; obtaining a preset neural network, and using the training sample set and the label corresponding to each training sample to train the preset neural network to obtain an image evaluation model.

[0224] In one embodiment, when the computer program is executed by a processor, the computer program performs the following steps: processing the first image using a preset image processing method to obtain a first target image; processing the second image using a preset image processing method to obtain a second target image; and subtracting the first target image from the second target image to obtain the detection image.

[0225] In one embodiment, when the computer program is executed by a processor, the computer program performs the following steps: using an oblique filter to extract an edge information map of the first image as a first target image; using an oblique filter to extract an edge information map of the second image as a second target image; and subtracting the second target image from the first target image to obtain an edge information loss map as the detection image.

[0226] In one embodiment, when the computer program is executed by a processor, the computer program performs the following steps: using a horizontal and vertical filter to extract an edge information map of the first image as a first target image; using a horizontal and vertical filter to extract an edge information map of the second image as a second target image; and subtracting the first target image from the second target image to obtain an edge information enhanced map as the detection image.

[0227] In one embodiment, when the computer program is executed by a processor, the computer program performs the following steps: performing absolute value processing on the detection image; and correspondingly, using at least one image evaluation model to identify the detection image after the absolute value processing.

[0228] In one embodiment, when the computer program is run by a processor, it performs the following: when it is determined that there is a first recognition result that does not meet a preset condition among the at least one recognition result, determining the cause of the image quality problem based on the first recognition result; determining an adjustment plan based on the determined cause of the image quality problem; the adjustment plan is used to adjust the plan for obtaining the second image by processing the first image.

[0229] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0230] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0232] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0233] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0234] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image and a second image; determining a detection image according to the first image and the second image; At least one image evaluation model is used to identify the test image to obtain at least one recognition result; each recognition result includes a score representing the quality of the test image; wherein, The determining of a detection image according to the first image and the second image includes: Processing the first image using a preset image processing method to obtain a first target image; Processing the second image using a preset image processing method to obtain a second target image; The first target image and the second target image are subtracted to obtain the detection image, wherein the detection image includes an edge information loss map and an edge information enhancement map, wherein the edge information loss map represents the oblique edge information loss of the image quality between the first image and the second image, and the edge information enhancement map represents the horizontal and vertical edge information enhancement of the image quality between the first image and the second image.

2. The method according to claim 1, characterized in that The method further includes: generating an image evaluation model; the generating the image evaluation model includes: Obtain at least one training sample; each of the training samples is obtained by subtracting a first training image from a second training image; each of the training samples has a corresponding label; Obtaining a training sample set according to the at least one training sample; A preset neural network is obtained, and the preset neural network is trained using the training sample set and the label corresponding to each training sample to obtain an image evaluation model.

3. The method according to claim 1, characterized in that Corresponding to the case where the image evaluation model is a fuzzy evaluation model, determining the detection image based on the first image and the second image includes: using an oblique filter to extract an edge information map of the first image as a first target image; Using an oblique filter to extract an edge information map of the second image as a second target image, the oblique filter is used for edge detection, and the Sobel operator of the oblique filter is an improvement on the Sobel operator of the horizontal and vertical filters; The first target image is subtracted from the second target image to obtain an edge information loss map as the detection image.

4. The method according to claim 1, wherein When the image evaluation model is a blockiness evaluation model, determining a detection image based on the first image and the second image includes: Using horizontal and vertical filters, extracting an edge information map of the first image as a first target image; Applying horizontal and vertical filters to extract an edge information map of the second image as a second target image; The first target image is subtracted from the second target image to obtain an edge information enhanced image as the detection image.

5. The method according to claim 3 or 4, characterized in that The method further comprises: Performing absolute value processing on the detection image; Accordingly, using at least one image evaluation model to identify the detection image includes: At least one image evaluation model is used to identify the detection image after absolute value processing.

6. The method according to claim 1, characterized in that The method further comprises: When it is determined that there is a first recognition result that does not meet the preset condition in the at least one recognition result, determining a cause of the image quality problem according to the first recognition result; An adjustment plan is determined based on the determined cause of the image quality problem; the adjustment plan is used to adjust a plan for obtaining the second image by processing the first image.

7. An image processing device, characterized in that: The device comprises: A first processing module, configured to acquire a first image and a second image; a second processing module, configured to determine a detection image based on the first image and the second image; The third processing module is configured to use at least one image evaluation model to identify the detection image and obtain at least one recognition result; each recognition result includes a score representing the quality of the detection image; wherein, The second processing module is used to process the first image using a preset image processing method to obtain a first target image; Processing the second image using a preset image processing method to obtain a second target image; The first target image and the second target image are subtracted to obtain the detection image, wherein the detection image includes an edge information loss map and an edge information enhancement map, wherein the edge information loss map represents the oblique edge information loss of the image quality between the first image and the second image, and the edge information enhancement map represents the horizontal and vertical edge information enhancement of the image quality between the first image and the second image.

8. The device according to claim 7, characterized in that The device further comprises: a pre-processing module for generating an image evaluation model; The preprocessing module is specifically used to obtain at least one training sample; each training sample is obtained by subtracting a first training image from a second training image; each training sample has a corresponding label; Obtaining a training sample set according to the at least one training sample; A preset neural network is obtained, and the preset neural network is trained using the training sample set and the label corresponding to each training sample to obtain an image evaluation model.

9. The device according to claim 7, characterized in that When the image evaluation model is a fuzzy evaluation model, the second processing module is specifically configured to use an oblique filter to extract an edge information map of the first image as a first target image; Using an oblique filter to extract an edge information map of the second image as a second target image, the oblique filter is used for edge detection, and the Sobel operator of the oblique filter is an improvement on the Sobel operator of the horizontal and vertical filters; The first target image is subtracted from the second target image to obtain an edge information loss map as the detection image.

10. The device according to claim 7, characterized in that When the image evaluation model is a blockiness evaluation model, the second processing module is specifically configured to use a horizontal and vertical filter to extract an edge information map of the first image as a first target image; Applying horizontal and vertical filters to extract an edge information map of the second image as a second target image; The first target image is subtracted from the second target image to obtain an edge information enhanced image as the detection image.

11. The device according to claim 9 or 10, characterized in that The second processing module is further configured to perform absolute value processing on the detection image; Correspondingly, the third processing module is used to use at least one image evaluation model to identify the detection image after absolute value processing.

12. The device according to claim 7, characterized in that The apparatus further includes: a fourth processing module, configured to, when determining that there is a first recognition result that does not meet a preset condition in the at least one recognition result, determine a cause of the image quality problem according to the first recognition result; An adjustment plan is determined based on the determined cause of the image quality problem; the adjustment plan is used to adjust a plan for obtaining the second image by processing the first image.

13. An image processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images