A thumbnail generation method based on salient object detection and image quality evaluation

Abstract

The application discloses a thumbnail generation method based on salient object detection and image quality evaluation, comprising the following steps: 1) preparing a salient object detection data set and training a model YOLO_SAL based on the data set; 2) inputting an image into the model YOLO_SAL to determine a salient core region of the image; 3) generating a to-be-screened set around the salient core region of the image through a cropping algorithm; and 4) screening out a thumbnail with the best aesthetic quality from the to-be-screened set through an image quality evaluation model SAMP_Net based on a composition rule. The application solves the problems of the existing thumbnail generation method based on deep learning, such as complex model, complicated steps, difficulty in landing and incomplete labeling target, etc. by combining YOLO_SAL and SAMP_Net, and improves the detection speed and the aesthetic quality of the thumbnail while ensuring that the predicted image core region has saliency and integrity.

Description

Technical Field

[0001] This invention relates to the technical field of thumbnail generation, and in particular to a thumbnail generation method based on salient target detection and image quality assessment. Background Technology

[0002] With the advent of the internet information age, the role of images in conveying information on news websites, e-commerce websites, and social media platforms has become increasingly prominent. Compared with text, voice, or video, images are more efficient in disseminating information due to their small size, intuitiveness, and rich content. Large websites and mobile applications such as TikTok and Weibo contain dozens or even hundreds of images or videos on each page. How to more effectively utilize screen space and reduce image transmission time has become a problem that needs to be solved. To address this, researchers have proposed the concept of thumbnails. A thumbnail is a scaled-down version of the original image, clearly expressing the content of the original image within a certain size and aspect ratio. Thumbnails have a wide range of uses, but traditional thumbnail generation methods based on seam cropping or machine learning are very time-consuming and produce poor-quality images. Therefore, researching efficient thumbnail generation methods is of great significance.

[0003] In recent years, the application of object detection and saliency detection in deep learning for image processing has become increasingly mature. Object detection models can accurately detect specific targets; saliency detection models can learn higher-level features through deep neural networks, making the extracted image content more representative. Applying object detection and saliency detection to thumbnail generation technology is extremely helpful in determining the cropping region of an image. Furthermore, current deep learning-based thumbnail generation methods typically generate multiple thumbnails at the model output, then use image quality assessment models to evaluate the image's content features or geometric relationships, ultimately selecting the one with the best aesthetic appeal. A complete thumbnail generation model requires combining all of the above methods, but due to the cumbersome steps and complex models, even recently proposed methods have not yet met the requirements for industrial application. Moreover, existing methods do not fully consider the specific characteristics of thumbnail generation tasks, resulting in the selected image quality not being optimal. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and propose a thumbnail generation method based on salient target detection and image quality assessment. By combining the bi-branch salient target detection model YOLO_SAL and the image quality assessment model SAMP_Net, this invention solves the problems of existing deep learning-based thumbnail generation methods, such as complex models, cumbersome steps, difficulty in implementation, and incomplete target annotation. While ensuring the saliency and integrity of the predicted image core region, this invention improves the detection speed and the aesthetic quality of the thumbnail.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a thumbnail generation method based on salient target detection and image quality assessment, comprising the following steps:

[0006] 1) Create a salient object detection dataset based on public datasets in the field of salient detection. The method is to label the smallest rectangular region in each importance map of the dataset, which contains more than 75% of the overall importance of the map, as the salient object region of that map.

[0007] 2) Construct a dual-branch salient object detection model YOLO_SAL. The model is an improved YOLOv5 network. The improvement is that a salient detection branch with an attention mechanism is added after the 5th layer of the basic module of the YOLOv5 network. The original structure of YOLOv5 is retained as the object detection branch. The salient detection branch is trained using the dataset created in step 1) and the object detection branch is trained using the COCO dataset. The improved YOLOv5 network has a significant improvement in accuracy and speed in detecting salient object regions.

[0008] 3) Input the image to be detected into YOLO_SAL. The object detection branch of YOLO_SAL detects all complete objects in the image, and the salient detection branch detects the salient object regions in the image. By combining the detection results of the two branches, the core region of the image is determined. A cropping algorithm based on image diagonal translation is designed. This algorithm is used to crop around the core region of the image to generate multiple images, forming a set to be screened.

[0009] 4) Feed the set to be screened into the image quality assessment model SAMP_Net, and select the images with the best aesthetic quality as the final generated thumbnails.

[0010] Furthermore, in step 1), the salient object detection dataset is created based on the publicly available datasets SALICON and DUTS. For each image, the publicly available datasets provide the original image and importance map, including categories such as people, animals, buildings, landscapes, and sports. The salient object regions that need to be labeled in the dataset are the smallest rectangular regions whose total importance in the image's importance map accounts for more than 75% of the total importance in the entire image. The formula is expressed as:

[0011]

[0012] In the above formula, let G be an importance map in a public dataset, p be a single pixel in map G, G(p) be the importance value of a single pixel, R be an arbitrary rectangular region in the image, and a given percentage τ represent the sum of the importance values ​​of all pixels within region R. The sum of the importance of all pixels in image G The proportion;

[0013] The method for labeling salient target regions is as follows: In G, find a rectangular region R with an importance score and percentage of the whole image τ of 0.75, an aspect ratio of 1, and the smallest area, and determine it as the salient target in the original image; use this method to label all importance maps contained in the datasets SALICON and DUTS, and use them as the salient target detection dataset.

[0014] Furthermore, step 2) includes the following steps:

[0015] 2.1) Design a two-branch salient object detection model, YOLO_SAL, based on the improved YOLOv5 network. YOLO_SAL retains the basic structure of YOLOv5 as the object detection branch and trains this branch using the COCO dataset. The other branch of YOLO_SAL is the salient detection branch introduced after the 5th base module. The two branches share the feature maps output by the first 5 base modules. The object detection branch divides the input image into N square grids, where the value of N depends on the resolution of the input image. Then, it predicts three metrics for each grid cell: bounding box, confidence score, and classification probability. In actual detection, it first determines whether the predicted confidence score of each bounding box exceeds a set threshold. If it does, it is considered that there is a target within the bounding box, thus obtaining the target's location. Then, it filters the bounding boxes containing targets using the non-maximum suppression algorithm, removing duplicate bounding boxes corresponding to the same target. Finally, based on the classification probabilities of the filtered bounding boxes, it takes the index corresponding to the highest probability, which is the target's classification index number, thus obtaining the target's category.

[0016] 2.2) The saliency detection branch adopts a feature pyramid network structure. The feature layers, from low to high, can detect salient targets of different sizes. This branch feeds the outputs of the 28th, 30th, and 33rd layers into different SE modules, and then concatenates them before feeding them into the saliency detection head. The SE module is an added attention mechanism module. The SE module performs global pooling on each feature map u through a squeeze operation, averaging it into a real number Z, which has a global receptive field. Specifically, the real number Z is calculated for the c-th channel of the feature map. c The formula is as follows:

[0017]

[0018] In the formula, H is the height of feature map u, W is the width of feature map u, and u c (i,j) represents the feature value of the c-th channel in row i and column j in feature map u;

[0019] The formula for calculating the real number Z is as follows:

[0020]

[0021] In the formula, c represents the c-th channel of the feature map, and Z is a real number. c The global feature value of the c-th channel is obtained by summing the feature values ​​of the three channels to obtain the real number Z, which is the global vector.

[0022] After the squeeze operation, the network outputs a feature map of size 1*1*c. Then, the excitation operation is used to learn the correlations between the c channels. The excitation operation is implemented by processing the global vector Z obtained in the previous step through two fully connected layers W1 and W2 to obtain the channel weight values ​​Q. After passing through the two fully connected layers, different values ​​in Q represent the weight information of different channels, assigning different weights to the channels. The formula for calculating Q is as follows:

[0023] Q=σ(W2δ(W1Z))

[0024] In the formula, δ is the ReLU activation function of the first fully connected layer W1, and σ is the Sigmoid activation function of the second fully connected layer W2.

[0025] Finally, the SE module treats the output of the excitation operation as the importance of each channel after feature selection, and combines it with the previous features through multiplicative weighting, thereby achieving the function of enhancing important features and suppressing unimportant features.

[0026] YOLO_SAL's salient detection branch is responsible for predicting the bounding boxes and confidence scores of salient targets, while the target detection branch is responsible for predicting the bounding boxes, confidence scores, and classifications of entity targets.

[0027] The loss function for the object detection branch is the same as that of the YOLOv5 network, while the saliency detection branch does not consider classification loss because it only predicts objects of one class. The loss function Loss for the saliency detection branch is calculated as follows:

[0028] Loss=λ1*LossObj+λ2*LossRECT

[0029] In the formula, LossObj is the confidence loss, which ranges from 0 to 1; λ1 is the confidence loss coefficient, which represents the credibility of the predicted bounding box. The larger the value, the greater the probability that the target exists in the bounding box; LossRECT is the bounding box loss, which represents the size and precise location of the target; and λ2 is the bounding box loss coefficient.

[0030] Further, in step 3), the image to be detected G1 is input to YOLO_SAL. Let the output of the salient detection branch of YOLO_SAL be the salient region set S, and the output of the target detection branch be the target region set T. The elements of set S and set T are arranged in descending order of confidence. S[0] represents the first element in set S, that is, the salient region with the highest confidence; T[0] represents the first element in set T, that is, the target region with the highest confidence. In order to ensure the salientity of the image core region and the integrity of the target, the formula for obtaining the image core region I is as follows, based on the existence and positional relationship of each rectangular box region in set S and set T:

[0031] a. When both sets S and T are empty, the formula is: I = Mid(G1), where the input of the Mid function is the image G1 to be detected, and the output is a rectangular area with the geometric center of G1 as the center and half the width and height of G1 as the width and height.

[0032] b. When set S is not empty and set T is empty, the formula is: I = S[0];

[0033] c. When set T is not empty and set S is empty, the formula is: I = T[0];

[0034] d. When neither set S nor set T is empty, take the salient region S[0] with the highest confidence in set S as the benchmark, traverse all entity targets in set T and calculate the intersection-union ratio (IOU) of the two rectangular box regions. When traversing to the nth entity target in set T, the following cases are discussed:

[0035] d1. When T[n] and S[0] are in an inclusive relationship, the formula is: I = Max(T[n], S[0]), where the input of the Max function is the rectangular area T[n] and S[0], and the output is the larger area of ​​the two areas.

[0036] d2. When T[n] and S[0] are intersecting, the formula is I = Union(T[n], S[0]), where the input of the Union function is the rectangular box region T[n] and S[0], and the output is the smallest rectangular box region in the image G1 to be detected that can completely contain both of them.

[0037] d3. When T[n] and S[0] are separated, the formula is: I = S[0].

[0038] Furthermore, based on the core region I of the image, a translation crop is performed along the two diagonals of the original image. During the translation operation, the offset set of the upper left corner of the rectangular frame region is set to [L, 3*L, 5*L], where L is 1 / 50 of the width of the rectangular frame, and the width and height offsets of the rectangular frame region are set to [2*L, 6*L, 10*L]. Subsequently, the cropped image is scaled to a specified resolution and combined into a thumbnail set ThumbNails, which is the set to be filtered.

[0039] Furthermore, in step 4), when selecting the best image from the set to be screened, the SAMP_Net image quality evaluation model based on composition rules is introduced. This model incorporates composition rules from the field of photography into the network model through multi-mode pooling.

[0040] Input the ThumbNails set to be screened in step 3) into SAMP_Net. The model evaluates the aesthetic quality of each image in the set based on the composition rules of the photography domain. The output of the model is a one-dimensional vector with the same length as the ThumbNails set. Each real value in the vector represents the aesthetic evaluation score of a single image. The image with the highest score is selected as the final generated thumbnail.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] 1. This invention generates a COCO format dataset suitable for object detection based on common saliency prediction datasets through annotation algorithms, transforming the computationally intensive saliency detection problem into a computationally less intensive object detection problem, which helps improve the efficiency of thumbnail generation.

[0043] 2. This invention designs a high-performance dual-branch salient object detection model, YOLO_SAL. This model is based on the classic object detection network YOLOv5, retaining the basic structure of YOLOv5 as the object detection branch, and separately designing a salient detection branch for predicting salient object regions. These two branches share the underlying feature extraction network and can be trained independently. By combining the outputs of the two branches, the salient core regions of the image are obtained, ensuring the saliency and integrity of the targets within the regions, and improving detection speed and accuracy.

[0044] 3. This invention designs a cropping algorithm based on the salient core region of an image. This algorithm fully considers the salientity of the core region of the image and generates an image set that satisfies various photographic composition rules through cropping according to certain rules. This makes the images in the set to be selected more comprehensive in terms of aesthetic quality distribution than previous cropping algorithms.

[0045] 4. This invention incorporates the latest advancements in image quality assessment and addresses the unique characteristics of thumbnail generation tasks. For the first time, it applies the SAMP_Net image quality assessment model, based on composition rules, to the thumbnail selection step. Compared to existing thumbnail generation techniques, this method can more accurately select the most aesthetically pleasing images from a set of images with essentially the same content by evaluating photographic composition rules. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method of the present invention.

[0047] Figure 2 This is a structural diagram of the YOLO_SAL bi-branch salient object detection model; where Image is the input image, Conv, C3, SPPF, Unsample, and Concat modules have the same structure as their counterparts in the YOLOv5 network, SE module is the introduced attention mechanism module, Detect is the detection head of the object detection branch, and SalDetect is the detection head of the salient detection branch.

[0048] Figure 3 The thumbnail generation effect diagram of the method of the present invention. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0050] like Figure 1 As shown, this embodiment provides a thumbnail generation method based on salient target detection and image quality assessment, including the following steps:

[0051] 1) Create a salient object detection dataset based on public datasets in the field of salient detection. The method is to label the smallest rectangular region in each importance map of the dataset that contains more than 75% of the overall importance of the map as the salient object region of that map.

[0052] The dataset was created based on the publicly available SALICON and DUTS datasets for salient object detection, including categories such as people, animals, buildings, landscapes, and sports. For each image, the original image and importance map are provided in the public datasets. The salient object regions to be labeled in the dataset are the smallest rectangular regions whose total importance in the image's importance map accounts for more than 75% of the total importance in the entire image. The formula is expressed as:

[0053]

[0054] In the above formula, let G be an importance map in a public dataset, p be a single pixel in map G, G(p) be the importance value of a single pixel, R be an arbitrary rectangular region in the image, and a given percentage τ represent the sum of the importance values ​​of all pixels within region R. The sum of the importance of all pixels in image G The proportion.

[0055] The method for labeling salient target regions is as follows: In G, find a rectangular region R with the smallest area, an importance score (τ) of 0.75, an aspect ratio of 1, and the smallest possible area, and define it as the salient target in the original image. Use this method to label all importance maps in the SALICON and DUTS datasets, and use them as the salient target detection dataset.

[0056] 2) Construct a two-branch salient object detection model, YOLO_SAL, which is an improved YOLOv5 network. The improvement involves adding a salient detection branch with an attention mechanism after the 5th layer of the basic YOLOv5 network, while retaining the original YOLOv5 structure as the object detection branch. The salient detection branch is trained using the dataset created in step 1), and the object detection branch is trained using the COCO dataset. The improved YOLOv5 network shows significant improvements in accuracy and speed in detecting salient object regions. This includes the following steps:

[0057] 2.1) A two-branch salient object detection model, YOLO_SAL, was designed based on the classic object detection network YOLO_v5. The structure is shown below. Figure 2 As shown, YOLO_SAL retains the basic structure of YOLOv5 as its object detection branch and trains this branch using the COCO dataset. The other branch of YOLO_SAL is the saliency detection branch, introduced after the 5th base layer. Both branches share the feature maps output from the first 5 base layers. The object detection branch divides the input image into N square grids, where N depends on the image resolution. For each grid cell, three metrics are predicted: bounding box, confidence score, and classification probability. During actual detection, the predicted confidence score of each bounding box is first checked against a set threshold. If it exceeds the threshold, the bounding box is considered to contain an object, thus determining the object's location. Next, a non-maximum suppression algorithm is used to filter bounding boxes containing objects, removing duplicate bounding boxes corresponding to the same object. Finally, based on the classification probabilities of the filtered bounding boxes, the index corresponding to the highest probability is taken as the object's classification index, thus determining the object's category.

[0058] 2.2) The saliency detection branch employs a feature pyramid network structure, with feature layers ranging from low to high to detect salient targets of different sizes. This branch feeds the outputs of layers 28, 30, and 33 into different SE modules, which are then concatenated before being fed into the saliency detection head. The SE module is an added attention mechanism module. Through a squeeze operation, the SE module performs global pooling on each feature map u, averaging it to a real number Z, which has a global receptive field. Specifically, the real number Z is calculated for the c-th channel of the feature map. c The formula is as follows:

[0059]

[0060] Where H is the height of feature map u, W is the width of feature map u, and u c (i,j) represents the feature value of the c-th channel in row i and column j in feature map u.

[0061] The formula for calculating the real number Z is as follows:

[0062]

[0063] Where c represents the c-th channel of the feature map, Z c The global eigenvalue of the c-th channel is obtained by accumulating the eigenvalues ​​of the three channels to obtain the real number Z, which is the global vector.

[0064] After the squeeze operation, the network outputs a feature map of size 1*1*c. Then, an excitation operation is used to learn the correlations between the c channels. The excitation operation is implemented by processing the global vector Z obtained in the previous step through two fully connected layers W1 and W2 to obtain the channel weight values ​​Q. After passing through the two fully connected layers, different values ​​in Q represent the weight information of different channels, assigning different weights to each channel. The formula for calculating Q is as follows:

[0065] Q=σ(W2δ(W1Z))

[0066] Where δ is the ReLU activation function of the first fully connected layer W1, and σ is the Sigmoid activation function of the second fully connected layer W2.

[0067] Finally, the SE module treats the output of the excitation operation as the importance of each channel after feature selection, and combines it with the previous features through multiplicative weighting, thereby achieving the function of enhancing important features and suppressing unimportant features.

[0068] YOLO_SAL's salient detection branch is responsible for predicting the bounding boxes and confidence scores of salient targets, while the target detection branch is responsible for predicting the bounding boxes, confidence scores, and classifications of entity targets.

[0069] The loss function for the object detection branch is the same as that of the YOLOv5 network, while the salient detection branch, which only predicts objects of one class, does not consider classification loss. The loss function (Loss) for the salient detection branch is calculated as follows:

[0070] Loss=λ1*LossObj+λ2*LossRECT

[0071] Among them, LossObj is the confidence loss, with a value ranging from 0 to 1. The confidence loss coefficient λ1 = 0.7. The confidence level represents the credibility of the predicted bounding box. The larger the value, the greater the probability that the target exists in the bounding box. LossRECT is the bounding box loss. The bounding box represents the size and precise location of the target. The bounding box loss coefficient λ2 = 0.05.

[0072] 3) Input the image G1 to be detected into YOLO_SAL. Let the output of the salient detection branch of YOLO_SAL be the salient region set S, and the output of the target detection branch be the target region set T. The elements of set S and set T are arranged in descending order of confidence. S[0] represents the first element in set S, that is, the salient region with the highest confidence; T[0] represents the first element in set T, that is, the target region with the highest confidence. In order to ensure the salience of the image core region and the integrity of the target, according to the existence and positional relationship of each rectangular box region in set S and set T, the formula for obtaining the image core region I has the following cases:

[0073] a. When both sets S and T are empty, the formula is: I = Mid(G1), where the input of the Mid function is the image G1 to be detected, and the output is a rectangular area with the geometric center of G1 as the center and half the width and height of G1 as the width and height.

[0074] b. When set S is not empty and set T is empty, the formula is: I = S[0];

[0075] c. When set T is not empty and set S is empty, the formula is: I = T[0];

[0076] d. When neither set S nor set T is empty, take the salient region S[0] with the highest confidence in set S as the benchmark, traverse all entity targets in set T and calculate the intersection-union ratio (IOU) of the two rectangular box regions. When traversing to the nth entity target in set T, the following cases are discussed:

[0077] d1. When T[n] and S[0] are in an inclusive relationship, the formula is: I = Max(T[n], S[0]), where the input of the Max function is the rectangular area T[n] and S[0], and the output is the larger area of ​​the two areas.

[0078] d2. When T[n] and S[0] are intersecting, the formula is I = Union(T[n], S[0]), where the input of the Union function is the rectangular box region T[n] and S[0], and the output is the smallest rectangular box region in the image G1 to be detected that can completely contain both of them.

[0079] d3. When T[n] and S[0] are separated, the formula is: I = S[0].

[0080] Based on the core region I of the image, the image is translated and cropped along the two diagonals of the original image. During the translation operation, the offset set of the upper left corner of the rectangular frame region is set to [L, 3*L, 5*L], where L is 1 / 50 of the width of the rectangular frame, and the width and height offsets of the rectangular frame region are set to [2*L, 6*L, 10*L]. Then, the cropped image is scaled to a specified resolution and combined into a thumbnail set ThumbNails, which is the set to be filtered.

[0081] 4) This method introduces the image quality assessment model SAMP_Net when screening images. SAMP_Net is the first model for image quality assessment based on composition rules. It integrates various photographic composition rules into the network model through multi-mode pooling. When combining with SAMP_Net, this method cancels the saliency map of the original model's default input and only inputs the original image into the model.

[0082] The ThumbNails set from step 3) is input into the SAMP_Net model. The model evaluates the aesthetic quality of each image in the set from the perspective of compositional rules in the field of photography. The model's output is a one-dimensional vector with the same length as the image set, where each real value represents the aesthetic score of a single image. This method selects the image with the highest score as the final generated thumbnail. The generated thumbnail effect is shown below. Figure 3 As shown.

[0083] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A thumbnail generation method based on salient target detection and image quality assessment, characterized in that, Includes the following steps: 1) Create a salient object detection dataset based on public datasets in the field of salient detection. The method is to label the smallest rectangular region in each importance map of the dataset that contains more than 75% of the overall importance of the map as the salient object region of that map. 2) Construct a dual-branch salient object detection model YOLO_SAL. The model is an improved YOLOv5 network. The improvement is that a salient detection branch with an attention mechanism is added after the 5th layer of the basic module of the YOLOv5 network. The original structure of YOLOv5 is retained as the object detection branch. The salient detection branch is trained using the dataset created in step 1), and the object detection branch is trained using the COCO dataset. 3) Input the image to be detected into YOLO_SAL. YOLO_SAL's object detection branch detects all complete objects in the image, and its salient detection branch detects salient object regions. By combining the detection results of the two branches, the core region of the image is determined. A cropping algorithm based on image diagonal translation is designed. This algorithm is used to crop multiple images around the core region of the image, forming a set to be filtered, as follows: Input the image G1 to be detected into YOLO_SAL. Let the output of the salient detection branch of YOLO_SAL be the salient region set S, and the output of the target detection branch be the target region set T. The elements of set S and set T are arranged in order of confidence from high to low. S[0] represents the first element in set S, that is, the salient region with the highest confidence. T[0] represents the first element in set T, i.e. the target region with the highest confidence; based on the existence and positional relationship of each rectangular region in sets S and T, the formula for obtaining the image core region I has the following cases: a. When both sets S and T are empty, the formula is: I = Mid(G1), where the input of the Mid function is the image G1 to be detected, and the output is a rectangular area with the geometric center of G1 as the center and half the width and height of G1 as the width and height. b. When set S is not empty and set T is empty, the formula is: I = S[0]; c. When set T is not empty and set S is empty, the formula is: I = T[0]; d. When neither set S nor set T is empty, take the salient region S[0] with the highest confidence in set S as the benchmark, traverse all entity targets in set T and calculate the intersection-union ratio (IOU) of the two rectangular box regions. When traversing to the nth entity target in set T, the following cases are discussed: d1. When T[n] and S[0] are contained within each other, the formula is: I = Max(T[n],S[0]), where the input of the Max function is the rectangular area T[n] and S[0], and the output is the larger area of ​​the two areas. d2. When T[n] and S[0] are intersecting, the formula is I = Union(T[n],S[0]), where the input of the Union function is the rectangular box region T[n] and S[0], and the output is the smallest rectangular box region in the image G1 to be detected that can completely contain both of them. d3. When T[n] and S[0] are separated, the formula is: I = S[0]; 4) Feed the set to be screened into the image quality assessment model SAMP_Net, and select the images with the best aesthetic quality as the final generated thumbnails.

2. The thumbnail generation method based on salient target detection and image quality assessment according to claim 1, characterized in that, In step 1), the salient object detection dataset is created based on the publicly available datasets SALICON and DUTS. For each image, the publicly available datasets provide the original image and importance map, including categories such as people, animals, buildings, landscapes, and sports. The salient object regions that need to be labeled in the dataset are the smallest rectangular regions whose total importance in the image's importance map accounts for more than 75% of the total importance in the entire image. The formula is expressed as: ； In the above formula, let a certain importance graph in the public dataset be... , For a single pixel in graph G, This represents the importance value of a single pixel. Given a percentage for any rectangular region in the image Representative area The sum of the importance of all pixels within. The sum of the importance of all pixels in graph G The proportion; The method for marking salient target regions is to find an importance level and a percentage of the entire map in region G. The smallest rectangular area with a length-to-width ratio of 0.75 and an aspect ratio of 1. Identified as a salient target in the original image; Using this method, all importance maps contained in the SALICON and DUTS datasets are labeled, and these datasets are used as salient object detection datasets.

3. The thumbnail generation method based on salient target detection and image quality assessment according to claim 2, characterized in that, Step 2) includes the following steps: 2.1) Design a two-branch salient object detection model, YOLO_SAL, based on the improved YOLOv5 network. YOLO_SAL retains the basic structure of YOLOv5 as the object detection branch and trains this branch using the COCO dataset. The other branch of YOLO_SAL is the salient detection branch introduced after the 5th base module. The two branches share the feature maps output by the first 5 base modules. The object detection branch divides the input image into N square grids, where the value of N depends on the resolution of the input image. Then, it predicts three metrics for each grid: bounding box, confidence score, and classification probability. In actual detection, it first determines whether the predicted confidence score of each bounding box exceeds a set threshold. If it does, it is considered that there is a target within the bounding box, thus obtaining the target's location. Then, it filters the bounding boxes containing targets using the non-maximum suppression algorithm, removing duplicate bounding boxes corresponding to the same target. Finally, based on the classification probabilities of the filtered bounding boxes, it takes the index corresponding to the highest probability, which is the target's classification index number, thus obtaining the target's category. 2.2) The saliency detection branch adopts a feature pyramid network structure. The feature layers, from low to high, can detect salient targets of different sizes. This branch feeds the outputs of layers 28, 30, and 33 into different SE modules, which are then concatenated before being fed into the saliency detection head. The SE module is an added attention mechanism module. The SE module performs a squeeze operation to globally pool each feature map u, averaging it to a real number Z, which has a global receptive field. Specifically, a real number is calculated for the c-th channel of the feature map. The formula is as follows: ； In the formula, H is the height of feature map u, and W is the width of feature map u. The feature value in the c-th channel, i-th row, j-th column of the feature map u; The formula for calculating the real number Z is as follows: ； In the formula, c represents the c-th channel of the feature map, and the real number... The global feature value of the c-th channel is obtained by summing the feature values ​​of the three channels to obtain the real number Z, which is the global vector. After the squeeze operation, the network outputs a feature map of size 1 * 1 * c. Then, the excitation operation is used to learn the correlations between the c channels. The excitation operation is implemented through two fully connected layers. , The globalized vector Z obtained in the previous step is processed to obtain the channel weight values. After two fully connected layers, Different values ​​in the text represent the weight information of different channels. Different weights are assigned to channels, and calculations are performed. The formula is as follows: ； In the formula, It is the first fully connected layer. The activation function ReLU, It is the second fully connected layer. The activation function is Sigmoid; Finally, the SE module treats the output of the excitation operation as the importance of each channel after feature selection, and combines it with the previous features through multiplicative weighting, thereby achieving the function of enhancing important features and suppressing unimportant features. YOLO_SAL's salient detection branch is responsible for predicting the bounding boxes and confidence scores of salient targets, while the target detection branch is responsible for predicting the bounding boxes, confidence scores, and classifications of entity targets. The loss function for the object detection branch is the same as that of the YOLOv5 network, while the saliency detection branch does not consider classification loss because it only predicts objects of one class. The loss function Loss for the saliency detection branch is calculated as follows: ； In the formula, The confidence loss takes a value between 0 and 1. The confidence loss coefficient represents the credibility of the predicted bounding box. The larger the value, the greater the probability that the target exists in the bounding box. The loss function is a rectangular bounding box, where the bounding box represents the size and precise location of the target. This represents the loss coefficient for the rectangular frame.

4. The thumbnail generation method based on salient target detection and image quality assessment according to claim 3, characterized in that, Based on the core region I of the image, the image is translated and cropped along the two diagonals of the original image. During the translation operation, the offset set of the upper left corner of the rectangular frame region is set to [L, 3*L, 5*L], where L is 1 / 50 of the width of the rectangular frame, and the width and height offsets of the rectangular frame region are set to [2*L, 6*L, 10*L]. Then, the cropped image is scaled to a specified resolution and combined into a thumbnail set ThumbNails, which is the set to be filtered.

5. The thumbnail generation method based on salient target detection and image quality assessment according to claim 4, characterized in that, In step 4), when selecting the best image from the set to be filtered, the SAMP_Net image quality evaluation model based on composition rules is introduced. This model incorporates composition rules from the field of photography into the network model through multi-mode pooling. Input the ThumbNails set to be screened in step 3) into SAMP_Net. The model evaluates the aesthetic quality of each image in the set based on the composition rules of the photography domain. The output of the model is a one-dimensional vector with the same length as the ThumbNails set. Each real value in the vector represents the aesthetic evaluation score of a single image. The image with the highest score is selected as the final generated thumbnail.

Images