Image redirection quality evaluation method and device based on reconstructed structural distortion

Abstract

The application provides an image retargeting quality evaluation algorithm based on structure distortion reconstruction. Information reconstruction is completed by inverse matching of a retargeting image to an original image, and structure distortion measurement indexes Sd and Vd based on global reconstruction information distribution and significant reconstruction information distribution are respectively proposed according to the distribution characteristics of the reconstruction information. The proposed structure distortion measurement indexes Sd and Vd are used in quality evaluation of different types of retargeting images, and the evaluation performance of the algorithm is verified through the public RetargetMe and CUHK databases, and the results show that the method has higher correlation and more stable evaluation performance. In addition, the proposed structure distortion measurement indexes Sd and Vd do not need to detect and compare structure information, thereby reducing algorithm errors and time consumption.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image quality evaluation, and more particularly relates to a method and device for evaluating the quality of image reorientation based on reconstructed structural distortion. BACKGROUND

[0002] During the image reorientation process, due to the size changes and content alternation operations such as information deletion and recombination, the phenomenon of image structure distortion occurs, which is referred to as structural distortion. Structural distortion is an important factor affecting the overall visual quality of the reoriented image. Effective measurement of structural distortion can quickly improve the performance of the image reorientation quality evaluation algorithm.

[0003] Taking image reduction as an example, most of the existing image reorientation algorithms [1-19] inevitably need to delete or reduce the display ratio of part of the image content, and the remaining content is recombined into a reoriented image. These methods mainly include the SC method based on line clipping [4-6] and the method based on warping [10-14] However, due to the large differences in the recombined content and its structure, there is a significant structural incoordination between the recombined information, which ultimately leads to a significant decrease in the visual quality of the reoriented image. As shown in Figure 1 is the image reorientation process of the salient region, in which the red part is the deleted content, and the remaining content is recombined during the reorientation process. However, there is a large difference between the remaining content and its structure, which ultimately leads to a significant deformation of the salient region of the reoriented image.

[0004] In addition, if the deleted information during the reorientation process is almost concentrated in a certain area, it will also destroy the overall structural symmetry of the reoriented image. As shown in Figure 2 , the person in the original image is in the center position of the image, while the person in the reoriented image is in the leftmost position, obviously the overall structural symmetry of the image is destroyed during the image reorientation process. However, the existing image reorientation quality evaluation algorithm [33-51] ignores the existence of this situation, and the evaluation index cannot match the perception characteristics of the human eye, thereby reducing the performance of the evaluation algorithm. SUMMARY

[0005] In order to solve the problems existing in the current structural distortion measurement method of the reoriented image, the present application proposes an image reorientation quality evaluation algorithm based on reconstructed structural distortion. This algorithm uses inverse matching to realize the information reconstruction of the reoriented image to the original image. According to the distribution of the reconstructed information, a simple mathematical calculation equation is designed to obtain the structural distortion score of the reoriented image. This method does not need to detect and compare the structural information, which not only ensures the accuracy of the structural distortion score, but also greatly reduces the time consumption of the distortion measurement.

[0006] To achieve the above object, according to one aspect of the present application, there is provided an image retargeting quality evaluation method based on structure distortion, comprising the following steps:

[0007] (1) Starting from the information reconstruction characteristics of the image retargeting process, two structure distortion measurement indexes based on the reconstruction information distribution are designed, i.e. a structure distortion measurement based on the significant reconstruction information distribution and a structure distortion measurement based on the global reconstruction information distribution, which measure the global structure symmetry destruction and the structure distortion of the significant area of the retargeted image respectively, and the two structure distortion measurements do not need to detect and compare the structure equal information, but only need to reconstruct the information from the retargeted image to the original image;

[0008] (2) The significant area structure distortion S d and the global structure symmetry distortion V d are used in the classification evaluation mechanism, S d is used to measure the significant area structure distortion of all types of retargeted images, in order to reflect that the selected significant area reconstruction information distribution range is enough to reflect the structure distortion of the important content, S d is weighted by using the weight ε, and V d is used to measure the structure symmetry destruction degree of all types of retargeted images, and different weights are set for the distortion scores V d of different types of retargeted images.

[0009] In one embodiment of the present application, in the step (1), the information reconstruction based on the reverse matching is firstly performed, the relative displacement of the matching feature points from the retargeted image to the original image is obtained, then the relative displacement is mapped to a two-dimensional graph with the same size as the original image to generate a reverse matching graph, and finally the global reconstruction information distribution graph and the significant area reconstruction information distribution graph are constructed by using the reverse matching information.

[0010] In one embodiment of the present application, the generation of the reverse matching graph specifically comprises: firstly, the matching feature points from the retargeted image to the original image are obtained by using the SIFT-Flow algorithm, then the relative displacement of the matching feature points is obtained, and then the relative displacement is mapped to a two-dimensional graph with the same size as the original image to generate a reverse matching graph, so as to realize the reverse matching from the retargeted image to the original image, in the reverse matching graph, the information that can be matched is called the non-deleted information, and the information that cannot be matched is the information deleted in the image retargeting process.

[0011] In one embodiment of the present application, the constructing the global reconstruction information distribution map specifically comprises: displaying the deleted information in the reverse matching map with black color, and displaying the remaining information not deleted with white color, and obtaining a binary map, that is, the map is called the global reconstruction information distribution map G c .

[0012] In one embodiment of the present application, the significant region reconstruction information distribution map specifically comprises:

[0013] The significant region of the original image is obtained, the significant region of the original image is detected by using a significant detection method PiCANe, and according to a pixel index matrix Id(x, y) of the white region, assuming that the significant region of the original image is a vector matrix of Mc rows and Nc columns, vertex coordinates of the significant region are determined as {(1, 1), (1, H s ), s , (W s , H s )}, that is, the significant region can be represented as {(x, y) | x [1, W s ], y [1, H s ]}. Four vertexes of the significant region of the original image are mapped to the global reconstruction information distribution map G c , and then a framed region is extracted from G c , and the region constitutes the significant region reconstruction information distribution map S c ,

[0014] S c = {G c (x, y) | x [1, W s ], y [1, H s ]} (1).

[0015] In one embodiment of the present application, the constructing the structure distortion measure based on the significant reconstruction information distribution specifically comprises:

[0016] Step 1: generating position vectors of rows and columns in the significant region reconstruction information distribution map, if the deleted information in the significant region reconstruction information distribution map takes a value of 0, and the remaining information takes a value of 1, then Sc can be represented as a corresponding 0-1 distribution matrix, and Ri and Ci represent the distribution vectors of the i-th row and the i-th column respectively, according to positions of “1” in the distribution vectors Ri and Ci, corresponding position vectors Ri_index and Ci_index are generated;

[0017] Step 2: calculating the structure distortion of rows and columns in the significant region, according to the position vectors Ri_index and Ci_index of the i-th row and the i-th column in the significant region reconstruction information distribution map, the structure distortion of the rows and the columns can be respectively calculated by using formula (2) and formula (3):

[0018]

[0019]

[0020] where mc, nc represent the number of remaining information of the ith row and the ith column respectively, assuming that the original image salient region is set as a matrix of Ws rows and Hs columns, then mc<=Ws, nc<=Hs, the reconstruction structure distortion set of all rows and columns is obtained by formula (2) and formula (3), denoted as Rd and Cd respectively, the average structure distortion of all rows and columns of the salient region when reconstructed is shown in formula (4) and (5):

[0021]

[0022]

[0023] Step 3: Measure the stability of the reconstructed information between rows and columns, the difference degree of the structure distortion between rows and columns is used to indicate the stability of the remaining information reconstruction process of the redirected image, denoted as vRd and vCd respectively, the calculation method is shown in formula (6) and (7):

[0024]

[0025]

[0026] When the values of vRd and vCd become larger, the difference of the structure distortion between rows and columns becomes larger, at this time, the distribution characteristics of the reconstructed information between rows and columns change greatly, the information reconstruction is in an unstable state, and then obvious structure distortion between rows and columns is prone to occur;

[0027] Step 4: Calculate the structure distortion of the salient region, integrate formula (4), (5), (6) and formula (7), obtain the overall structure distortion of the salient region of the image in the redirection process, denoted as Sd:

[0028] S d = a x mR d + b x mC d + c x vR d + d x vC d (8)

[0029] where the first two terms of formula (8) represent the structure distortion generated when the information of all rows and columns is reconstructed; the last two terms represent the stability of the information reconstruction between rows and columns, the weights of the four distortion indexes are a, b, c, d respectively, the larger Sd is, the greater the content difference between the reconstructed information is, and the more serious the structure distortion of the redirected image is.

[0030] In one embodiment of the present application, the structure distortion metric based on the global reconstruction information distribution includes:

[0031] Step 1: Invert the global reconstruction information distribution map G d , and set the deleted information as 1 and the reserved information as 0, as shown in equation (9): c

[0032] G d =1-G c (9)

[0033] Step 2: Set the region measuring the global structure symmetry distortion, and set H and W to represent the height and width of the global reconstruction information distortion map G d , the global structure symmetry distortion mainly comes from the number of whole row / column deleted information, in order to accurately locate the direction of whole row / column deleted information aggregation, set the horizontal axis through the center coordinate (H / 2, W / 2) of the global reconstruction information distribution map to divide the global reconstruction information into two parts, and set the vertical axis through the center coordinate to divide the global reconstruction information distortion map into two parts;

[0034] Step 3: Obtain the deleted information distribution vectors in the horizontal and vertical directions, the deleted information distribution vector of the i-th row in the upper half part is R i , and the corresponding information deletion amount is V ud (i), as shown in equation (10), through the value range of i [1, H / 2], the statistics of the whole row information deletion amount of all rows in the upper half part are obtained, that is, the structure symmetry distortion in the upper direction, as shown in equation (11):

[0035]

[0036]

[0037] Similarly, the structure symmetry distortions in the lower direction, the left direction and the right direction are obtained, as shown in equations (12), (13) and (14) respectively:

[0038]

[0039]

[0040]

[0041] Step 4: Determine the structure symmetry distortion of the redirected image, the number of whole row information deletion in the upper and lower parts obtained in step 3 is mV ud and mV bd , and the number of whole column information deletion in the left and right parts is mV ld and mV rd ​This determines the amount of primary information removed in the vertical and horizontal directions, which serves as a score for symmetry distortion in both directions, denoted as V. dv and V dh Finally, the largest amount of information deletion in both the vertical and horizontal directions is selected as the structural distortion measure V based on the global reconstruction of information distribution. d As shown in equation (17):

[0042] V dv =max(mV ud mV bd (15)

[0043] V dh =max(mV ld mV rd (16)

[0044] V d =max(mV dv mV dh (17)

[0045] V d The larger the value of , the more severe the destruction of the global structural symmetry of the redirected image.

[0046] In one embodiment of the present invention, in step (2), ε represents the boundary of the reconstructed information within the salient region, and v r It is the ratio of the salient region width to the original image width; while u r As the ratio of the salient region height to the original image height, ε = 2 × v r ×u r / (v r 2 +u r 2 ).

[0047] In one embodiment of the present invention, in step (2):

[0048] The quality evaluation method for SSR-type retargeted images is as follows:

[0049] Q SSR =Q S +Q L +λ×(1-SEGS)+(λ+ε)×S d +w ssr ×V d (18)

[0050] Among them, Q S Q represents the geometric distortion measure score. Ldenotes the information loss measure score, SEGS denotes the similarity of color structure improved by using significant region, and the role is to measure the distortion degree of color texture, S d denotes the structure distortion based on the distribution of reconstructed information, the weight is (λ+ε), ε denotes the boundary of reconstructed information in the significant region, and λ denotes the proportion of the significant region in the image;

[0051] The image quality evaluation of the LSR type is: Q LSR = Q S + Q L + λ × (1-SEGS) + (λ+ε) × S d + w lsr × V d ; w lsr of the LSR type is 0, and a more effective significant information loss indicator Q L is selected to measure the distortion of the image of this type;

[0052] The image quality evaluation of the MSR type is: Q MSR = Q S + Q L + λ × (1-SEGS) + (λ+ε) × S d + w msr × V d ; w msr is set for the image of the MSR type;

[0053] The image quality evaluation of the NSR type is: Q NSR = (1-BARS) + (1-BIL) + w nsr × V d .

[0054] According to another aspect of the present application, there is also provided an image redirection quality evaluation device based on reconstructed structure distortion, comprising at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the image redirection quality evaluation method based on reconstructed structure distortion after being executed by the processor.

[0055] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0056] First, a global structure symmetry distortion measure index based on reconstructed information is proposed in the method. The index effectively evaluates the global structure symmetry destruction degree of the redirection image by using the distribution of reverse matching information, aiming at the phenomenon of a large amount of whole row or whole column information loss in the redirection image;

[0057] Secondly, a structure distortion measurement index based on the distribution of the salient reconstruction information is proposed in the method. The index obtains the salient reconstruction information by using the salient region of the original image and the inverse matching information, and then accurately measures the structure distortion of the salient region of the retargeted image according to the distribution characteristics of the salient reconstruction information.

[0058] Thirdly, the results of the experiment show that, compared with the existing methods, the objective evaluation results generated by the method have higher correlation with the subjective evaluation results, and the evaluation performance is more stable.

[0059] Fourthly, when evaluating the structure distortion of the retargeted image, the method does not need to detect and compare the structure information as the existing methods do, thereby reducing the dependence on the accuracy of the structure feature detection algorithm and reducing the complexity of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 FIG. 1 is a schematic diagram of the image retargeting process of the salient region in the embodiment of the present application;

[0061] Figure 2 FIG. 2 is a schematic diagram of the symmetry destruction process of the retargeted image in the embodiment of the present application;

[0062] Figure 3 FIG. 3 is a schematic diagram of the inverse matching process in the embodiment of the present application;

[0063] Figure 4 FIG. 4 is a schematic diagram of the extraction process of the global reconstruction information distribution graph and the salient region reconstruction information distribution graph in the embodiment of the present application;

[0064] Figure 5 FIG. 5 is a schematic diagram of the structure distortion measurement process based on the salient reconstruction information distribution in the embodiment of the present application;

[0065] Figure 6 FIG. 6 is a schematic diagram of the vertical direction structure symmetry distortion measurement process in the embodiment of the present application;

[0066] Figure 7 FIG. 7 is a schematic diagram of the horizontal direction structure symmetry distortion measurement process in the embodiment of the present application;

[0067] Figure 8 FIG. 8 is a schematic diagram of the KRCC value of 37 groups in the RetargetMe database in the embodiment of the present application;

[0068] Figure 9 FIG. 9 is a schematic diagram of the time consumption comparison of the structure distortion measurement in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0070] 1. Structure distortion metric based on reconstructed information distribution

[0071] The present application designs two structure distortion metric indexes based on reconstructed information distribution from the information reconstruction characteristics of the image redirection process, which respectively measure the global structure symmetry destruction of the redirected image and the structure distortion phenomenon of the salient region. The two structure distortion metrics proposed do not need to detect and compare the structure equal information, but only need to perform information reconstruction of the redirected image to the original image. The following three subsections respectively introduce the specific calculation process of the information reconstruction process, the two structure distortion metric indexes based on the salient region and the global reconstructed information distribution.

[0072] 1.1 Information reconstruction based on reverse matching

[0073] The present application learns from the reverse matching algorithm

[35] , first obtains the relative displacement of the matching feature points from the redirected image to the original image, then maps the relative displacement to a two-dimensional graph with the same size as the original image to generate a reverse matching graph, and finally uses the reverse matching information to construct the global reconstructed information distribution graph and the salient region reconstructed information distribution graph. The specific steps are as follows:

[0074] Step 1: Generate a reverse matching graph. First, use the SIFT-Flow algorithm

[28] to obtain the matching feature points from the redirected image to the original image, as shown in Figure 3 for the reverse matching graph generation process. Then, obtain the relative displacement of the matching feature points, and then map the relative displacement to a two-dimensional graph with the same size as the original image to generate a reverse matching graph, thereby realizing the reverse matching from the redirected image to the original image. In the reverse matching graph, the information that can be matched is called non-deleted information, and the information that cannot be matched is the information deleted in the image redirection process.

[0075] Step 2: Construct a global reconstructed information distribution graph. As shown in Figure 4 for the extraction of the global reconstructed information distribution graph and the salient region reconstructed information distribution graph, the deleted information in the reverse matching graph is displayed in black, and the remaining non-deleted information is displayed in white, to obtain a binary graph, which is called the global reconstructed information distribution graph G c .

[0076] Step 3: Obtain the salient region of the original image. As shown in Figure 4 , the salient region of the original image is detected by using the saliency detection method PiCANet

[21] , and the vertex coordinates of the salient region are determined as {(1, 1), (1, H s ), (W s , 1), (W s , H s )} according to the pixel index matrix Id(x, y) of the white region, that is, the salient region can be represented as {(x, y) | x ∈ [1, W s ], y ∈ [1, H s ]}.

[0077] Step 4: Extract the salient region reconstruction information distribution map. Map the four vertices of the salient region of the original image to the global reconstruction information distribution map G c , and then extract the framed region from G c , which constitutes the salient region reconstruction information distribution map S c .

[0078] S c = {G c (x, y) | x ∈ [1, W s ], y ∈ [1, H s ]} (1)

[0079] 1.2 Structure distortion measure based on salient reconstruction information distribution

[0080] Based on the characteristics of the salient region reconstruction information distribution, the present application designs a structure distortion measurement method based on the salient region reconstruction information distribution, Figure 5 as shown in the structure distortion measurement framework diagram based on the salient reconstruction information distribution. The method includes four steps, which are as follows:

[0081] Step 1: Generate the position vectors of the rows and columns in the salient region reconstruction information distribution map. If the deleted information (black) in the salient region reconstruction information distribution map is 0, and the value of the remaining information (white) is 1, then Sc can be represented as the corresponding 0-1 distribution matrix. Let Ri and Ci represent the distribution vectors of the i-th row and the i-th column, respectively, and according to the positions of "1" in the distribution vectors Ri and Ci, generate the corresponding position vectors Ri_index and Ci_index, as shown in Figure 5 .

[0082] Step 2: Calculate the structure distortion of rows and columns in the salient region. According to the position vectors Ri_index and Ci_index of the i-th row and i-th column in the salient region reconstruction information distribution map, the structure distortion of the row and column can be calculated respectively by using formula (2) and formula (3):

[0083]

[0084]

[0085] wherein mc and nc represent the number of residual information of the i-th row and i-th column respectively, assuming that the salient region of the original image is set as a matrix of Ws rows and Hs columns, then mc <= Ws and nc <= Hs. The set of reconstruction structure distortion of all rows and columns can be obtained by formula (2) and formula (3). Denoted as Rd and Cd respectively. Therefore, the average structure distortion of all rows and columns in the salient region when reconstructed is shown in formula (4) and (5):

[0086]

[0087]

[0088] Step 3: Measure the stability of the reconstruction information between rows and columns. The difference degree of the structure distortion between rows and columns is used to measure the stability of the residual information reconstruction process of the redirected image, denoted as vRd and vCd respectively, and the calculation method is shown in formula (6) and (7):

[0089]

[0090]

[0091] When the values of vRd and vCd become larger, it means that the difference of the structure distortion between rows and columns becomes larger, at this time, the reconstruction information distribution characteristics between rows or columns change greatly, and the information reconstruction is in an unstable state, then obvious structure distortion between rows and columns is easy to appear.

[0092] Step 4: Calculate the structure distortion of the salient region. By integrating formula (4), (5), (6) and formula (7), the overall structure distortion of the salient region in the redirection process is obtained, denoted as Sd:

[0093] S d = a x mR d + b x mC d + c x vR d + d x vC d (8)

[0094] where the first two terms in equation (8) represent the structural distortion caused by the information reconstruction of all rows and columns; the last two terms represent the stability of the information reconstruction between rows and between columns. The weights of the four distortion indicators are a, b, c, d. In the present application, since the four distortions have the same impact on the quality of the redirected image, a = b = c = d = 0.25 is set. The greater Sd is, the greater the content difference between the reconstructed information is, and the more serious the structural distortion of the redirected image is.

[0095] 1.3 Structural distortion measure based on global reconstructed information distribution

[0096] The present application considers that the phenomenon of a large number of single-sided whole rows or columns of information deletion is a serious destruction of the overall structural symmetry of the image, and designs a structural distortion measure method based on global reconstructed information distribution, Figure 6 which is a measure of vertical direction structural symmetry distortion, Figure 7 which is a measure of horizontal direction structural symmetry distortion. It accurately reflects the information deletion of whole rows or columns in the redirection process, and aims to improve the performance of the evaluation algorithm.

[0097] The specific calculation process of the proposed structural symmetry distortion measure method based on global reconstructed information distribution is as follows:

[0098] Step 1: Invert the global reconstructed information distribution map G d . Set the deleted information in the global reconstructed information distribution map G c to 1, and the remaining information to 0, as shown in equation (9):

[0099] G d = 1 - G c (9)

[0100] Step 2: Set the area to measure the global structural symmetry distortion. Let H and W represent the height and width of the global reconstructed information distortion map G d , respectively. The global structural symmetry distortion mainly comes from the number of whole row / column deleted information. In order to accurately locate the direction of whole row / column deleted information aggregation, set a horizontal axis (red line) through the center coordinate (H / 2, W / 2) of the global reconstructed information distribution map as a base point to divide the global reconstructed information into two parts, as shown in Figure 6 ; set a vertical axis (yellow line) through the base point to divide the global reconstructed information distortion map into two parts, as shown in Figure 7 .

[0101] Step 3: Obtain the deleted information distribution vectors in the horizontal and vertical directions. As shown in Figure 6 , the deleted information distribution vector of the i-th row in the upper half is R i , and the corresponding information deletion amount is V ud(i), as shown in equation (10). By the range of i [1, H / 2], the statistics of the whole row information deletion of the upper half part are obtained, i.e. the structural symmetry distortion in the upward direction, as shown in equation (11).

[0102]

[0103]

[0104] Similarly, the structural symmetry distortion in the downward direction, the left direction and the right direction are obtained, as shown in equations (12), (13) and (14) respectively.

[0105]

[0106]

[0107]

[0108] Step 4: Determine the structural symmetry distortion of the redirected image. The number of information deletion of the whole row in the upper and lower parts obtained in step 3 is mV ud and mV bd , and the number of information deletion of the whole column in the left and right parts is mV ld and mV rd , so as to determine the main information deletion in the vertical and horizontal directions as the symmetry distortion measurement scores in the two directions, denoted as V dv and V dh . Finally, the maximum information deletion in the vertical and horizontal directions is selected as the structural distortion measure V d based on the global reconstruction information distribution, as shown in equation (17):

[0109] V dv = max (mV ud , mV bd ) (15)

[0110] V dh = max (mV ld , mV rd ) (16)

[0111] V d = max (mV dv , mV dh ) (17)

[0112] The greater the value of V d , the more serious the global structural symmetry destruction of the redirected image.

[0113] 2 Image redirection quality evaluation algorithm based on reconstruction structural distortion

[0114] To improve the efficiency of the existing image reorientation quality evaluation algorithm to evaluate structural distortion, the algorithm of the application uses the proposed significant region structural distortion S d and global structural symmetry distortion V d For the classification evaluation mechanism, an image reorientation quality evaluation algorithm based on reconstruction structural distortion is proposed. Specifically, the algorithm first uses the proposed S d To measure the significant region structural distortion of all types of reoriented images, in order to reflect that the selected significant region reconstruction information distribution range is sufficient to reflect the structural distortion of important content, the S d is weighted by weight ε, which represents the boundary of the reconstruction information in the significant region, and v r is the ratio of the width of the significant region to the width of the original image; and u r is the ratio of the height of the significant region to the height of the original image. Therefore, ε = 2 × v r × u r / (v r 2 + u r 2 Secondly, since the human eye has different perceptions of structural distortion of different types of reoriented images, the algorithm of the application uses V d to measure the degree of destruction of structural symmetry of all types of reoriented images, and sets different weights for the distortion scores V d of different types of reoriented images.

[0115] 2.1 Other distortion measurement indicators

[0116] Since the reoriented image may have geometric distortion, structural distortion and information loss at the same time, in addition to the structural distortion measurement indicators, the algorithm of the application also introduces geometric distortion and information loss related measurement indicators, so that the evaluation algorithm is consistent with the subjective evaluation, which helps to improve the performance of the algorithm. As shown in Table 1, some distortion measurement indicators used when the algorithm of the application and the SDC-IRQA algorithm based on significant driving classification image reorientation objective quality evaluation algorithm (SDC-IRQA) are used to evaluate:

[0117] Table 1 Distortion metrics

[0118]

[0119] As can be seen from Table 1, the SDC-IRQA algorithm lacks significant region based structural distortion and information loss measurement indicators, and the algorithm of the application supplements the corresponding structural distortion and information loss measurement indicators of the reoriented image. Compared with the SDC-IRQA algorithm, the distortion factors used by the algorithm of the application are more comprehensive, the distortion measurement indicators can be selected more, and the performance of the image reorientation quality evaluation can be better improved.

[0120] 2.2 Different types of redirected image quality assessment

[0121] (1) SSR type redirected image quality assessment

[0122] There is a small area of salient region in SSR type image, but when the salient region is too small, the geometric distortion of the salient object is easy to occur. Therefore, in order to effectively measure the quality of SSR type redirected image, three kinds of geometric distortion are selected for redirected quality measurement as in SDC-IRQA algorithm; in addition, the information loss of salient region is an important reason for quality degradation in human visual perception [35-39] ; and when the salient region of SSR type redirected image is too small, the content distortion near the salient region needs to be considered. The quality assessment method of SSR type redirected image is given in equation (18):

[0123] Q SSR = Q S + Q L + λ × (1-SEGS) + (λ+ε) × S d + w ssr × V d (18)

[0124] Wherein, Q S represents the geometric distortion measurement score, Q L represents the information loss measurement score, and specific as equation 19 and equation 20. SEGS represents the similarity of color structure improved using salient region

[36] , which measures the distortion degree of color texture. S d represents the structure distortion based on the distribution of reconstructed information, and the weight is (λ+ε), ε represents the boundary of reconstructed information in the salient region, and λ represents the proportion of salient region in the image.

[0125] Q S = (1-BARS) + λ × (1-SARS) + (1-FARS) (19)

[0126]

[0127] V d represents the global structure damage caused by a large amount of information deletion, and in addition, V d is different for the damage degree of different types of salient region in geometry, structure and other contents. The foreground F

[51] is obtained by using the saliency detection method O , and then the salient region detection method

[21] is used on F OThe salient object extraction is performed to obtain a larger salient region S composed of a representative salient region in the SSR image and the content near the salient region l S is calculated using equation (21) l The aspect ratio similarity w ssr of S is used as the weight of V d , and aims to evaluate the distortion of the salient region and the content near the salient region. The calculation method of w ssr is as follows:

[0128]

[0129] wherein the larger salient region of the redirected image S l is obtained by matching S l using the salient region extraction method of the redirected image represents the ratio of the width of S t to the width of the original image, and H r represents the ratio of the corresponding height. M r represents the average value of W r and H r , and K and a are respectively 1e-6 and 0.3, which are used to avoid the abnormality of the left and right sides.

[0130] (2) LSR class redirected image quality evaluation

[0131] Geometric distortion, information loss and other indicators are also needed in the LSR class to comprehensively evaluate the quality, but since the size of the salient region in the LSR class image is almost equal to the size of the image, the area where distortion occurs is always within the salient region, and V d measures a large number of whole row / column losses, which belong to the loss of salient information. Therefore, in order to reduce the redundancy of the evaluation indicators, w lsr in the LSR class is 0, and a more effective salient information loss indicator Q L is selected to measure the distortion of the redirected image in this class.

[0132] Q LSR = Q S + Q L + λ × (1-SEGS) + (λ+ε) × S d + w lsr × V d (22)

[0133] (3) MSR class redirected image quality evaluation

[0134] In MSR-type images with multiple salient regions, besides considering the geometric distortion and information loss of the salient regions in the retargeted image, more attention is paid to the structural harmony between the multiple salient regions. Since these multiple salient regions are often distributed in a dispersed manner, the area involved is quite large. Therefore, distortion V... d The existence of this feature can, in severe cases, directly delete certain prominent areas, disrupting the arrangement of multiple prominent areas.

[0135] Q MSR =Q S +Q L +λ×(1-SEGS)+(λ+ε)×S d +w msr ×V d (twenty three)

[0136] Therefore, weights w are set for MSR class images. msr It is necessary to explain how the deletion of entire rows or columns of information disrupts the harmony between salient areas. This invention's reference algorithm...

[40] The sum of the relative displacements of salient regions in the original and repositioned images is calculated to indicate the reconstructed distances between multiple important objects after repositioning, thus measuring the degree of coordination between these salient regions. Therefore, w msr Set as:

[0137]

[0138] Where (u,v) are the x and y coordinates of the relative displacement from the retargeted image to the original image obtained through inverse matching. m ,v m ) are the average values ​​of the x and y coordinates of all relative displacements, respectively.

[0139] (4) Quality evaluation of NSR-type retargeted images

[0140] Q NSR =(1-BARS)+(1-BIL)+w nsr ×V d (25)

[0141] The phenomenon of deleting entire rows or columns of information causes incomplete edges in NSR-type retargeted images without significant objects, resulting in distortions that are of visual interest. Algorithms are used to address this.

[36] The edge group similarity index (EGS) proposed in [the paper] serves as a measure of image distortion V in NSR (non-significant object) classes. d weight w nsr By combining BARS and BIL to measure the geometric distortion and information loss of retargeted images, the structural, geometric, and informational distortions of NSR-type retargeted images can be comprehensively measured.

[0142] Further, the present application also provides an image retargeting quality evaluation device based on reconstruction structure distortion, comprising at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the image retargeting quality evaluation method based on reconstruction structure distortion after being executed by the processor.

[0143] 3 Experimental results and discussion

[0144] The present application designs a reconstruction information distribution based structure distortion measurement index S which does not rely on structure information detection and comparison d and V d , and proposes an image retargeting quality evaluation algorithm based on reconstruction structure distortion. In order to verify the effectiveness of the proposed S d and V d measure the structure distortion of the retargeted image, and demonstrate the universality of the proposed evaluation algorithm, the present application also uses RetargetMe

[20] and CUHK

[22] two databases to test the performance of the evaluation algorithm.

[0145] 3.1 RetargetMe database evaluation performance

[0146] As shown in Table 2, the test results of the evaluation algorithm on the RetargetMe dataset show the performance comparison results of the present application evaluation algorithm and the existing 13 image retargeting quality evaluation algorithms, including BDS

[25] , EH

[26] , EMD

[29] , PGDIL

[33] , BARS

[35] , BNSSD

[40] , PYMO

[48] , MLF

[36] , CDM

[23] , FGM

[41] , DeEval

[51] , RPCS

[50] and SDC-IRQA algorithm. First, the overall performance of the algorithm is shown, and the proposed method obtains an average KRCC value of 0.654. Compared with other algorithms, the correlation of the present application algorithm with the subjective evaluation has been greatly improved. Specifically, the average KRCC value of the present application algorithm is higher than that of the image retargeting quality evaluation algorithm which also measures the structure distortion of the retargeted image

[36] ,

[40] and

[41] The performance is improved by 27.7%, 38.3% and 18.3%, which is 12.2% higher than the SDC-IRQA algorithm.

[0147] In addition, the performance comparison data of the algorithm of the application in the image evaluation of the six attributes of Line Edge, Faces People, Foreground Object, Texture, Geometric Structure and Symmetry are also given in Table 3. It can be seen that the average KRCC values obtained by the algorithm of the application are significantly higher than those of other algorithms. Specifically, first, for the attributes of Line Edge, Faces / People, Foreground Object, Geometric Structure and Symmetry, the correlation of the algorithm of the application is much greater than 0.6. Second, for the evaluation of the Line / Edge attribute class redirection image, the average KRCC value of the algorithm of the application is 26.1% higher than the latest achievement FGM

[41] of the traditional method, which is 12.2% higher than the latest achievement RPCS

[50] of the deep learning method.Furthermore, for the quality evaluation of the Geometric / Structure attribute redirection image, the average KRCC value of the algorithm of the application is 0.683, while the average KRCC of the algorithm

[36] ,

[40] and

[41] , which also focuses on structural distortion, is only 0.536, 0.497 and 0.554. In addition, the algorithm of the application obtains an average KRCC of 0.619 for the Symmetry attribute, which is 30.0% higher than the average KRCC of the RPCS

[50] , which also measures the degree of destruction of image symmetry.

[50]

[0148] The excellent performance of the algorithm of the application is due to the two structural distortion measurement indicators S d and V d based on the information distribution of the reconstruction. The S d indicator measures the structural distortion according to the discordance phenomenon caused by the residual information of the salient region, avoids the information mismatch phenomenon existing in the structural information detection and comparison process, and thus guarantees the accuracy of the evaluation algorithm. By showing the relationship between the information distribution and the structural distortion in the salient region redirection process, the algorithm of the application is more consistent with the subjective analysis of the source of structural distortion, and thus the performance of the evaluation algorithm is improved. V dThen, the symmetry destruction of the retargeted image is measured from the phenomenon of the loss of a large amount of information sensitive to the human eye, and different weights are set for different types of retargeted images, so as to further distinguish the distortion characteristics of different retargeted images, so that each type of retargeted image is evaluated most appropriately, thereby improving the overall performance of the evaluation algorithm.

[0149] Table.2 Test results on RetargetMe database

[0150] Table.2 Test results on RetargetMe database

[0151]

[0152] Table.3 Test results of S d and V d Evaluation performance test results

[0153] Table.3 Test results of S d and V d

[0154]

[0155] Table 4 Performance on four sets

[0156] Table 4 Performance on four sets

[0157]

[0158]

[0159] In addition, in order to show the effectiveness of the proposed structure distortion measurement index S d and V d based on the distribution of the reconstructed information, the present application removes the S d and V d part of the quality score in the final distortion measure combined score Q, and is denoted as Q1. As shown in Table 3, without S d and V d , the average KRCC value obtained by Q1 is 0.593, and when S d is added to Q1 for combination, the KRCC value can reach 0.620; and when Q1 is combined with V d , the average KRCC value can reach 0.616. By observing the average KRCC value corresponding to the Symmetry attribute in Table 3, it can be seen that the effectiveness of V d in evaluating the image structure destruction phenomenon, and when V dThe average KRCC value corresponding to the Symmetry attribute is only 0.571, while the Q1 combination V d The average KRCC value corresponding to the Symmetry attribute is 0.607, which is 6.4% higher than the KRCC value when Q1 is evaluated alone.

[0160] Table.5 Ranking statistics of top 20KRCC value on RetargetMe database

[0161] Table.5 Ranking statistics of top 20KRCC value on RetargetMe database

[0162]

[0163] The algorithm uses V d The different weights of V d The evaluation effect of the algorithm on different types of salient regions is shown in Table 4. It can be seen from the data that the algorithm is more excellent than the SDC-IRQA algorithm. In particular, for the evaluation of LSR type of large area salient region, the average KRCC value of the algorithm is 0.804, which is 28.6% higher than the SDC-IRQA algorithm. This is because the proposed index V d The amount of information deletion is effectively measured, and the important content distortion of the LSR type is effectively measured, so the evaluation of the LSR type image is more related to the subjective evaluation.

[0164] Figure 8 The KRCC values of 37 image groups in the RetargetMe database are shown in Table 5. In addition, the average KRCC values of the top 5, top 10, top 15 and top 20 among the KRCC values of the 37 image groups are shown in Table 5. Compared with other algorithms, the average KRCC values of the top 5, top 10, top 15 and top 20 of the algorithm are all greater than 0.8. Figure 8 The high correlation values in the algorithm are more than those in the algorithm based on the edge structure

[36] and the SDC-IRQA algorithm using classification evaluation, and the performance of the algorithm is the best.

[0165] The reasons for the best performance of the algorithm are: first, because the algorithm

[36] The algorithm measures global structural distortion of the retargeted image by edge group similarity, but lacks a metric for structural distortion in salient regions. Furthermore, the algorithm...

[36] Structural detection algorithms are required. [31-32] Separately detecting the structural information of the original image and the reconstructed image cannot guarantee a match between the detection results, leading to errors in the evaluation score and reducing the correlation between the algorithm and subjective evaluation. This invention addresses this by designing a significant structural distortion metric S based on reconstructed information. d and V d Starting from the subjectively sensitive phenomenon of incoordination in the distribution of reconstructed information, this invention directly predicts the distortion of image structure by observing the characteristics of the reconstructed information distribution, without requiring structure detection and comparison processes. The algorithm comprehensively measures the structural distortion, geometric distortion, and information loss in the global and salient regions of the retargeted image, improving the correlation between objective evaluation algorithms and subjective evaluations.

[0166] 3.2 Performance Evaluation of CUHK Database

[0167] Similar to the SDC-IRQA algorithm, this invention also uses the CUHK database to test the correlation between the proposed objective evaluation algorithm for image repositioning quality based on reconstructed structural distortion and the subjective evaluation. To demonstrate the evaluation effect on different types of repositioned images, as shown in Table 6, the algorithm of this invention and the structural distortion-based MLF algorithm...

[36] Compared to the SDC-IRQA algorithm, the data shows that when evaluating LSR images, the average PLCC value of the algorithm in this invention is as high as 0.849, which is significantly higher than that of the MLF algorithm.

[36] It is 11% and 18.7% higher than the SDC-IRQA algorithm, respectively. The average SRCC value reaches 0.828, which is higher than the MLF algorithm.

[36] The SDC-IRQA algorithm showed a growth of 14.5% and 17.2%, respectively. This is mainly because, when evaluating LSR-type images, the algorithm in this invention focuses on measuring distortion in large salient regions of these images, compared to the MLF algorithm.

[36] In comparison, the algorithm adds a salient region color structure distortion metric, SEGS, specifically for evaluating the quality of structure distortion in salient regions, while the MLF algorithm...

[36] The method only uses a global content-based structural distortion metric, which cannot effectively measure the structural distortion of salient objects. Compared to the SDC-IRQA algorithm, the NSR-type images fitted in this invention show slightly higher PLCC and SRCC values. This is because this invention incorporates the MLF algorithm when evaluating this type of image.

[36] The structural distortion metric EGS proposed in the application comprehensively measures the geometry, structure and information loss of the NSR image of the non-significant region class, thereby improving the correlation with the subjective evaluation. In addition, the RMSE and OR of the algorithm of the application remain minimum in the evaluation of the SSR and MSR images, thereby proving the stability of the algorithm evaluation.

[0168] Table.6 Performance of four sets on CUHK database

[0169] Table.6 Performance of four sets on CUHK database

[0170]

[0171] In order to reflect the overall evaluation performance of the algorithm of the application, the evaluation performance indicators PLCC, SRCC, RMSE and OR of the four types of redirected image quality are summarized into overall average performance indicators mPLCC, mSRCC, mRMSE and mOR by formula (21) and formula (22) as the SDC-IRQA algorithm, and compared with the existing 15 kinds of evaluation algorithms, as shown in Table 7. From the data, it can be seen that the mPMSE value of the algorithm of the application is 8.012, which indicates that the algorithm evaluation stability is the strongest. The average linear correlation value mPLCC and the rank correlation value mSRCC are 0.762 and 0.733 respectively, which are also higher than those of other redirected image quality evaluation algorithms. The reason why the evaluation performance of the algorithm of the application is better than the existing 15 kinds of evaluation algorithms is mainly two aspects: on the one hand, the classification evaluation mechanism is used in the algorithm of the application, and the best distortion evaluation indicator is used for each type of redirected image, so that each type of redirected image can have a better fitting correlation; on the other hand, the classification evaluation method used in the algorithm of the application is different from that of the SDC-IRQA algorithm, mainly in the form of weight to indicate the distortion difference of different significant regions, and combined with the proposed significant structure distortion and global structure distortion, the structure weight of different types of redirected images is set.

[0172] Table.7 Performance on CUHK database

[0173] Table.7 Performance on CUHK database

[0174]

[0175] In order to prove the effectiveness of the proposed structural distortion metric S d and Vd, Table 8 shows the evaluation performance of the redirected image of the significant region class in the CUHK database. Q2=Q S +QL + λ x (1-SEGS) represents the other common distortion combination of the reoriented image final evaluation combination except S d and V d . From Table 8, it can be found that the PLCC value of the distortion measurement combination without adding S d and V d is only 0.676, and when S d is added, the PLCC value can be improved to 0.763, and the performance improvement is 12.9%; the SRCC value is improved by 14.0%; the RMSE representing the stability of the evaluation is reduced by 7.1%, and the outlier rate data is also reduced from 0.103 to 0.095. In the fourth combination in Table 8, Q2 considers S d and V d at the same time, the PLCC value is directly improved from 0.676 to 0.781, with an improvement of 15.5%, and the outlier rate is reduced to 49.5% of the original. Not only is it more consistent with subjective evaluation, but it also ensures better evaluation stability. This is because the distortion measurement index of Q2 combination only focuses on measuring the reorientation quality on the outline of the salient region, and cannot explain the incoordination phenomenon of the specific content distortion inside the salient region. The index S d proposed by the present application estimates the content structure incoordination degree of the reoriented image through the abnormal phenomenon of information reconstruction process, and V d takes the different distortion phenomena caused by different types of salient regions due to a large amount of information deletion as the weight, simplifies the SDC-IRQA algorithm, and clearly defines the common distortion and different distortion of different types of reoriented images. The overall measurement based on structural distortion, geometric distortion and information loss improves the correlation between the objective evaluation algorithm and the subjective evaluation.

[0176] Table 8 Evaluation effectiveness of S d and V d

[0177] Table.8 Evaluation effectiveness of S d andV d

[0178]

[0179] 3.3 Time consumption comparison of structural distortion measurement method

[0180] The biggest difference between the traditional structural distortion measurement index and the S d and V d ​Neither content nor structural feature extraction process and feature comparison calculation is needed, only simple mathematical calculation on the distribution of reconstruction information is needed, and the consumption of computer is very small. In order to illustrate the time-saving advantage of the proposed S d and V d , Figure 9 The total time consumed by structural distortion measurement of 256 retargeted images in the RetargetMe database is counted, and the proposed structural distortion measurement index S d is compared with two existing structural distortion measurement methods SEGS

[35] and IR-SSIM

[34] . The running environment is a 64-bit dual-core 3.20GHz processor PC, and the tools are Matlab, C++, etc. It can be seen from Figure 9 that the time consumed by the proposed S d index is only 18.28 seconds, the time consumed by SEGS

[35] is 2940.48 seconds, and the time consumed by IR-SSIM

[34] is 41974.64 seconds. The time consumed by the proposed S d index is much smaller than that of SEGS

[35] and IR-SSIM

[34] .

[0181] The proposed structural distortion measurement index can greatly save the calculation time of structural distortion score, because S d does not need to perform structural feature extraction process and comparison process, but requires the computer to perform simple mathematical calculation on the reconstruction information map, thereby greatly reducing the time consumption of the algorithm.

[0182] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

[0183] References

[0184] [1] H. Liu, X. Xie, W. Y. Ma, and H. Zhang. Automatic browsing of large pictures on mobile devices [C]. Proceedings of Eleventh ACM International Conference on Multimedia, 2003, pp. 148-155

[0185] [2] P. Krahenbuhl, M. Lang, A. Hornung, et al., A system for retargeting of streaming video [J], ACM Transactions on Graphics, 2009, p. 126.

[0186] [3] A. Santella, M. Agrawala, D. DeCarlo, D. Salesin, M. Cohen. Gaze-based interaction for semi-automatic photo cropping [C]. Proceedings of the SIGCHI conference on Human Factors in computing systems, 2006, pp. 771-780.

[0187] [4] Rahman Z, Pu Y F, Aamir M, et al. A framework for fast automatic image cropping based on deep saliency map detection and gaussian filter [J]. International Journal of Computers and Applications, 2019, 41(3): pp. 207-217.

[0188] [5] M. Rubinstein, A. Shamir, and S. Avidan, Improved seam carving for video retargeting [J]. ACM Transactions on Graphics, 2008, 27(3): pp. 16.

[0189] [6] Li C, Hu R, Liang C, et al. Faster Seam Carving for Video Retargeting [C].

[0190] Proceedings of IEEE International Conference on Image Processing, Athens, 2018,

[0191] pp. 823-827.

[0192] [7] Z. Karni, D. Freedman, C. Gotsman, Energy-based image deformation [J], Computer Graphics Forum, 2009, pp. 1257-1268.

[0193] [8] P. Yael, K. V. Eitam, P. Shmuel, Shift-map image editing [C], Proceedings of IEEE

[0194] 12th International Conference on Computer Vision, 2009, pp. 151-158.

[0195] [9] Kopf S, Guthier B, Hipp C, et al. Warping-based video retargeting for stereoscopic video [C] Proceedings of IEEE International Conference on Image Processing. 2015,

[0196] pp. 2898-2902.

[0197]

[10] L. Wolf, M. Guttmann, D. Cohen-Or, Non-homogeneous content-driven video retargeting [C]. Proceedings of the IEEE 11th International Conference on Computer Vision, 2007, pp. 1-6.

[0198]

[11] B. Li, Y. Chen, J. Wang, L. Y. Duan, and W. Gao. Fast retargeting with adaptive grid optimization [C]. Proceedings of IEEE International Conference on Multimedia and Expo,

[0199] 2011, pp. 1-4.

[0200]

[12] D. Panozzo, O. Weber, and O. Sorkine. Robust image retargeting via axis-aligned deformation[J]. Computer Graphics Forum, 2012, 31(2ptl): pp. 229-236.

[0201]

[13] C. H. Chang. A line-structure-preserving approach to image resizing [C]. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 1075-1082.

[14] P. Krahenbuhl, M. Lang, A. Hornung, M. Gross, A system for retargeting of streaming video[J]. ACM Transactions on Graphics, 2009, pp. 126.

[0202]

[15] M. Rubinstein, A. Shamir, and S. Avidan. Multi-operator media retargeting[J]. ACM Transactions on graphics, 2009, 28(3): pp. 23.

[0203]

[16] W. Dong, N. Zhou, J. C. Paul, et al., Optimized image resizing using seam carving and scaling[J], ACM Transactions on Graphics 28(5) (2009) 1-10.

[0204]

[17] Cho D, Park J, Oh T, Tai Y, Kweon I-S (2017) Weakly-and Self-Supervised Learning for Content-Aware Deep Image Retargeting[C]. Proceedings of IEEE International Conference on Computer Vision, pp 4568-4577.

[0205]

[18] Tan W, Yan B, Lin C, Niu X (2020) Cycle-IR: Deep Cyclic Image Retargeting [J]. IEEE Transactions on Multimedia 22(7): 1730-1743.

[0206]

[19] Y. Liang, Y. Liu and D. Gutierrez (2017) Objective Quality Prediction of Image Retargeting Algorithms [J]. IEEE Transactions on Visualization and Computer Graphics,

[0207] vol. 23, no. 2, pp. 1099-1110.

[0208]

[20] Retargetme. a benchmark for image retargeting, http: / / people.csail.

[0209]

[21] Y. Wang, C. Tai, O. Sorkine, and T. Lee, Optimized scale-and-stretch for image resizing [C]. ACM Transactions on Graphics, 2008, pp. 1.

[0210]

[22] L. Ma, W. Lin, C. Deng, K. N. Ngan, Image retargeting quality assessment: A study of subjective scores and objective metrics [J], IEEE Journal of Selected Topics in Signal Processing 6(6) (2012) 626-639.

[0211]

[23] Chen Zhibo, Lin Jianxin. Image retargeting quality evaluation method based on statistical similarity and bidirectional saliency fidelity:,

[0212] CN106447654B [P]. 2019.

[0213]

[24] C. L. Zitnick and P. Dollar, Edge boxes: Locating object proposals from edges, in Computer Vision [J], vol. 8693. Zurich, Switzerland: Springer, 2014, pp. 391-405.

[25] C. Hsu, C. Lin, Y. Fang, et al., Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss [J], IEEE Journal of Selected Topics in Signal Processing 8(3) (2014) 377-389.

[0214]

[26] Y. Zhang, Y. Fang, W. Lin, X. Zhang, Li, L. Backward registration-based aspect ratio similarity for image retargeting quality assessment [J]. IEEE Transactions on Image Processing, 2016, 25(9): pp. 4286-4297.

[0215]

[27] C. Liu, J. Yuen, and A. Torralba, SIFT Flow: Dense correspondence across scenes and its applications [J], IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp.

[0216] 978-994, May 2011.

[0217]

[28] A. Liu, W. Lin, H. Chen, et al., Image retargeting quality assessment based on support vector regression [J], Signal Processing: Image Communication 39 (2015)

[0218] 444-456. [1] F Afshar, Mansouri A, Zandi M. Image Retargeting Quality Assessment using Structural Similarity and Information Preservation Rate [C] 10th Iranian Conference on Machine Vision and Image Processing. 2017.

[0219]

[29] Y. Liang, Y. Liu, D. Gutierrez, Objective quality prediction of image retargeting algorithms [J], IEEE Transactions on Visualization and Computer Graphics 23(2) (2017)

[0220] 1099-1110.

[0221]

[30] Y. Zhang, W. Lin, Q. Li, et al., Multiple-level feature-based measure for retargeted image quality [J], IEEE Transactions on Image Processing 27(1) (2018) 451-463

[0222]

[31] Y. Zhang and K. N. Ngan, Objective quality assessment of image retargeting based on line distortion [C], 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary, 2016, pp.002505-002510.

[0223]

[32] P. Dollar and C. L. Zitnick, Structured forests for fast edge detection [C], in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2013, pp. 1841-1848.

[0224]

[33] S. Qi, Y. T. Chi, A. Peter, et al, Casair: Content and shape-aware image retargeting and its applications [J], IEEE Transactions on Image Processing 25(5) (2016)

[0225] 2222-2232.

[0226]

[34] Y. Zhang, K. N. Ngan. Region-based image retargeting quality assessment [C].

[0227] Proceedings of IEEE International Conference on Image Processing, 2015, pp. 1757-1761.

[0228]

[35] Z. Chen, J. Lin, N. Liao, et al., Full reference quality assessment for image retargeting based on natural scene statistics modeling and bidirectional saliency similarity [J], IEEE Transactions on Image Processing 26(11) (2017) 5138-5148.

[0229]

[36] B. Zhang, P. V. Sander and A. Bermak, Registration based retargeted image quality assessment [C], 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 2017, pp. 1258-1262.

[0230]

[37] Y. Li, L. Guo, L. Jin, A Content-Aware Image Retargeting Quality Assessment Method Using Foreground and Global Measurement [J]. IEEE Access. Vol. 7, pp. 91912-23, 2019.

[0231]

[38] B. Jiang, J. Yang, Q. Meng, B. Li and W. Lu, A Deep Evaluator for Image Retargeting Quality by Geometrical and Contextual Interaction [J], IEEE Transactions on Cybernetics, 2017, vol. 50, no. 1, pp. 87-99.

[0232]

[39] Y.-J. Liu, Y. Han, Z. Ye and Y.-K. Lai, Ranking-Preserving Cross-Source Learning for Image Retargeting Quality Assessment [J], IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020., vol. 42, no. 7, pp. 1798-1805.

[0233]

[40] Wu, Zhi-Shan, Zhang, Shuai, and Niu, Yu-Zhen. Image retargeting quality assessment based on multi-scale distortion perception features [J]. Journal of Beijing University of Aeronautics and Astronautics 45.12 (2019): 2487-2494.

[0234]

[41] Fu Z, Shao F, Jiang G, Yu M. Quality assessment of retargeted images combined with bidirectional similarity transformation[J]. Journal of Image and Graphics, 2018, 23(4): 0490-0499

[0235]

[42] Jiangyang Zhang and C.-C. Jay Kuo. 2014. An Objective Quality of Experience (QoE) Assessment Index for Retargeted Images. In Proceedings of the 22nd ACM international conference on Multimedia (MM'14). Association for Computing Machinery, New York, NY, USA, 257-266.

[0236]

[42] Jiangyang Zhang and C.-C. Jay Kuo. 2014. An Objective Quality of Experience (QoE) Assessment Index for Retargeted Images. In Proceedings of the 22nd ACM international conference on Multimedia (MM'14). Association for Computing Machinery, New York, NY, USA, 257-266.

[0237]

[43] M. Cheng, G. Zhang, N. J. Mitra, et al., Global contrast based salient region detection[J], IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3) (2011) 409-416.

[0238]

[44] N. Liu, J. Han, M. Yang, Picanet: Learning pixel-wise contextual attention for saliency detection[C], in: IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3089-3098.

[0239]

[45] Q. Hou, M. Cheng, X. Hu, et al., Deeply supervised salient object detection with short connections[C], in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5300-5309.

[0240]

[46] Z. Luo, A. Mishra, A. Achkar, et al., Non-local deep features for salient object detection [C], in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6593-6601.

[0241]

[47] Face++ research toolkit, www.faceplusplus.com (2013).

[0242]

[48] J. Harel, C. Koch, P. Perona, Graph-based visual saliency [C], in: Conference on Advances in Neural Information Processing Systems, 2006, pp. 545-552

[0243]

[49] M. G. Kendall, A new measure of rank correlation [J], Biometrika 30(1) (1938) 81-93.

[50] W. Dong, N. Zhou, J. C. Paul, et al., Optimized image resizing using seam carving and scaling [J], ACM Transactions on Graphics 28(5) (2009) 1-10.

[0244]

[51] Z. Karni, D. Freedman, C. Gotsman, Energy-based image deformation [J], in: Computer Graphics Forum, 2009, pp. 1257-1268.

Claims

1. An image quality evaluation method for image retargeting based on reconstruction distortion, characterized in that, Comprising the following steps: (1) from the information reconstruction characteristics of the image redirection process, two structure distortion measurement indexes based on the reconstruction information distribution are designed, namely the structure distortion measurement based on the significant reconstruction information distribution and the structure distortion measurement based on the global reconstruction information distribution, which respectively measure the global structure symmetry destruction and the structure distortion phenomenon of the significant area of the redirected image, the two structure distortion measurements do not need to detect and compare the structure information, only need to perform information reconstruction from the redirected image to the original image; in the step (1), first, the information reconstruction based on the reverse matching is performed, the relative displacement of the matching feature points from the redirected image to the original image is obtained, then the relative displacement is mapped to a two-dimensional graph with the same size as the original image, a reverse matching graph is generated, finally, the global reconstruction information distribution graph and the significant area reconstruction information distribution graph are constructed by using the reverse matching information; The construction of the structure distortion measurement based on the significant reconstruction information distribution specifically comprises: Step 1: generate the position vectors of the rows and columns in the significant area reconstruction information distribution graph, if the deleted information in the significant area reconstruction information distribution graph is 0 and the remaining information is 1, then Sc is represented as the corresponding 0-1 distribution matrix, Ri and Ci represent the distribution vectors of the ith row and the ith column respectively, according to the positions of "1" in the distribution vectors Ri and Ci, the corresponding position vectors Ri_index and Ci_index are generated; Step 2: calculate the structure distortion of the rows and columns in the significant area, according to the position vectors Ri_index and Ci_index of the ith row and the ith column in the significant area reconstruction information distribution graph, the structure distortion of the rows and columns is calculated respectively by using formula (2) and formula (3): Where mc and nc represent the first and second digits, respectively. row and number The remaining information in each column, assuming the salient region of the original image is set as a matrix of Ws rows and Hs columns, then mc<=Ws, nc<=Hs. The set of reconstructed structural distortions for all rows and columns is obtained from equations (2) and (3), denoted as Rd and Cd respectively. The average structural distortions generated when reconstructing all rows and columns of the salient region are shown in equations (4) and (5): Step 3: measure the stability degree of the reconstruction information between the rows and columns, the difference degree of the structure distortion between the rows and the columns is used to indicate the stability degree of the remaining information reconstruction process of the redirected image, which are respectively denoted as vRd and vCd, the calculation methods are as shown in formula (6) and (7): When the values of vRd and vCd become larger, the difference of the structure distortion between the rows and the columns becomes larger, at this time, the reconstruction information distribution characteristics between the rows or the columns change greatly, the information reconstruction is in an unstable state, and then the obvious structure distortion between the rows and the columns is prone to occur; Step 4: calculate the structure distortion of the significant area, by comprehensively using formula (4), (5), (6) and formula (7), the overall structure distortion of the significant area of the image in the redirection process is obtained, which is denoted as Sd: In formula (8), the first two terms represent the structure distortion generated when all the rows and columns are reconstructed; the last two terms represent the stability degree of the reconstruction of the rows and the columns, and the weights of the four distortion indexes are respectively a, b, c and d, the larger Sd is, the greater the content difference between the reconstruction information is, and the more serious the structure distortion of the redirected image is; The structure distortion measurement based on the global reconstruction information distribution comprises: Step 1: Inverted global reconstruction information profile The deleted information in the global reconstruction information profile is set to 1 and the reserved information is set to 0 as shown in equation (9): Step 2: Define the region for measuring the distortion of global structural symmetry. H and W These represent the global reconstruction information distortion maps. The distortion of the height and width of the global structure mainly stems from the amount of information deleted from entire rows / columns. To accurately locate the direction of the clustering of deleted information in entire rows / columns, the center coordinates of the globally reconstructed information distribution map are determined. Using the base point as the reference point, a horizontal axis is set through the base point to divide the global reconstruction information into upper and lower parts, and a vertical axis is set through the base point to divide the global reconstruction information distortion map into left and right parts. Step 3: Obtain the horizontal and vertical distribution vectors of the deleted information, the upper part of which is the first... i The distribution vector of the deletion information of the row is R i The corresponding amount of information deleted is V ud ( i As shown in equation (10), through i The range of values ​​is [1, ... H / 2], to obtain the statistics of the total number of rows deleted in the upper half, that is, the structural symmetry distortion in the upper direction, as shown in equation (11): Similarly, the structure symmetry distortion of the downward direction, the left direction and the right direction is obtained, which is respectively shown in formula (12), (13) and (14): Step 4: Determine the structural symmetry distortion of the redirected image. The number of information deletions in the upper and lower parts of the whole row obtained in step 3 is and , and the number of information deletions in the left and right parts of the whole column is and , so as to determine the main information deletion amount in the vertical and horizontal directions as the symmetry distortion measurement score of the two directions, denoted as and , and finally select the maximum information deletion amount in the vertical and horizontal directions as the structural distortion measure based on the global reconstruction information distribution , as shown in equation (17): The larger the value of , the more severe the destruction of the global structural symmetry of the redirected image; (2) Distortion of the structure of salient regions S d and global structure symmetry V d For the classification evaluation mechanism, first S d For measuring the distortion of the structure of salient regions of all types of redirected images, in order to reflect that the selected salient region reconstruction information distribution range is sufficient to reflect the structure distortion of important content, the weight is added to S d , and V d For measuring the degree of destruction of the structure symmetry of all types of redirected images, and setting different weights for the distortion scores V d of different types of redirected images.

2. The image-reorientation quality evaluation method based on reconstructed structural distortion according to claim 1, wherein, The generating the reverse matching map specifically comprises: firstly, obtaining matching feature points from the reoriented image to the original image by using a SIFT-Flow algorithm; then, obtaining relative displacements of the matching feature points; and then, mapping the relative displacements to a two-dimensional map with the same size as the original image to generate the reverse matching map, thereby realizing reverse matching from the reoriented image to the original image; in the reverse matching map, information that can be matched is referred to as non-deleted information, and information that cannot be matched is information deleted in the image reorientation process.

3. The image-reorientation quality evaluation method based on reconstructed structural distortion according to claim 1, wherein, The constructing global reconstruction information distribution map specifically comprises: displaying the deleted information with black color and the remaining information which is not deleted with white color in the reverse matching map, and obtaining a binary map, that is, the map is called a global reconstruction information distribution map .

4. The image-reorientation quality evaluation method based on reconstructed structural distortion according to claim 1, wherein, The significant region reconstruction information distribution map specifically comprises: obtaining a salient region of the original image, detecting the salient region of the original image by using a saliency detection method PiCANe, and determining a pixel index matrix of the white region according to the salient region , assuming that the salient region of the original image is a vector matrix of Mc rows and Nc columns, determining the vertex coordinates of the salient region as , that is, the salient region can be expressed as ; mapping the four vertexes of the salient region of the original image to a global reconstruction information distribution map , and then extracting a framed region from , which constitutes a salient region reconstruction information distribution map , 5. The image-renderring-quality-evaluation method based on reconstruction distortion according to claim 1, wherein, In the step (2), representing the boundary of the reconstructed information within the salient region, set is the ratio of the salient region width to the original image width; and as the ratio of the salient region height to the original image height, .

6. The image-reorientation quality evaluation method based on reconstructed structural distortion according to claim 1, wherein, In the step (2): The quality evaluation mode of the SSR type reoriented image is: wherein, denotes a geometric distortion measure score, denotes an information loss measure score, denotes a similarity using the color structure improved by salient regions, which aims to measure the distortion degree of color texture, S d denotes a structure distortion based on the distribution of reconstructed information, with a weight of , denotes the boundary of reconstructed information within salient regions, denotes the proportion of salient regions in the image; LSR class of redirected image quality evaluation is: ; LSR class of the middle value is 0, and the more effective significant information loss index measure the distortion of this class of redirected image; MSR class redirect image quality evaluation is: ; Set weights for MSR class images; NSR-based redirection image quality evaluation is: . 7.An image reorientation quality evaluation device based on reconstruction structure distortion, characterized in that: The device comprises at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the image reorientation quality evaluation method based on reconstruction structure distortion in any one of claims 1-6 after being executed by the processor.

Images