An image cropping method and device based on semantic perception

Abstract

The application discloses a kind of based on semantic perception's image cropping method and device, which effectively makes up the limitation of traditional method only relying on visual features, by introducing concept dictionary to enrich semantic information;Meanwhile, the present application further excavates potential semantic relationship through semantic similarity, which is helpful to fully exploit and utilize semantic information, so that a more rich and unified concept space is established between image and semantic label, further improving the accuracy of cropping result and the robustness of model;In addition, the present application breaks through the limitation of fixed angle of traditional method, by comprehensively considering the angle information of each instance, dynamically adjusting the direction of cropping frame, and by comprehensively considering the geometric properties, direction information, density distribution and other features of instance, automatically selecting appropriate cropping mode and composition ratio, so that the cropping result can more flexibly and accurately adapt to the layout and structure of image content.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image cropping, and particularly relates to an image cropping method and device based on semantic perception. BACKGROUND

[0002] The image cropping task aims to accurately cut the boundaries of the original image to retain the most informative and visually appealing parts of the image, thereby optimizing content display and enhancing visual effects. With the continuous development of digital image processing technology and the increasing diversification and individualization of image processing needs, image cropping has gradually become a key task in computer vision, automated design and other fields, and has been widely applied to social media, advertising design, image retrieval and other scenarios.

[0003] At present, most mainstream image cropping methods are based on image detection information, salient regions and other visual features, combined with composition principles and photography theories for cropping. However, many traditional methods ignore the deep semantic information contained in the image in the application process, failing to fully exploit the potential content of the image. At the same time, existing cropping techniques mostly use fixed cropping frame angles and proportions, without considering the diversity of instances and complexity of layout in the image, resulting in a lack of personalization and flexibility in the cropping results. How to organically combine the visual features and semantic information of the image and flexibly adjust the cropping method according to the specific content is still a key problem to be solved in the field of image cropping. SUMMARY

[0004] The purpose of the present application is to provide an image cropping method and device based on semantic perception.

[0005] In a first aspect, the present application provides an image cropping method based on semantic perception, comprising the following steps:

[0006] Step one, obtaining an original image ; using a segmentation detection model to obtain a pixel mask and a semantic label corresponding to each instance in the original image ; obtaining a feature joint representation from the pixel mask and the semantic label ; wherein, ; is the number of instances in the original image ;

[0007] Step two, taking the instances in the original image as nodes to construct a weighted undirected graph and an adjacency matrix ;

[0008] Step three, extracting the pixel mask set of boundary points ; through rotation matrix Set of boundary points Rotate to different angles to obtain the rotation angle. Rotation angle Make the set of boundary points The area of ​​the positively rotated rectangular boundary is minimized.

[0009] Step 4: Use a graph convolutional network to jointly represent features and weighted undirected graph Construct the Laplacian matrix; use spectral clustering to segment the eigenvalues ​​and eigenvectors of the Laplacian matrix, and use each cluster obtained from the segmentation as a group of candidate centroids;

[0010] Step 5: Based on the weights of the nodes in each candidate centroid group and rotation angle Determine the corresponding cutting angle ;in, ; For the first The number of nodes in the candidate centroid group; ; The number of candidate centroid groups; based on the cropping angle. Get coverage of the The smallest bounding rectangle of all nodes in the candidate centroid group is used as the smallest bounding box, and the final candidate boxes are obtained through a mapping method; the final candidate boxes are then used to refine the original image. Perform cropping to obtain the cropping result.

[0011] Preferably, in step one, the feature joint representation The method to obtain it is as follows:

[0012] Obtain each semantic tag using a concept dictionary Detailed definition ; using semantic tags and the corresponding definition The set formed as text input Use image encoder pixel mask Local visual features Use a text encoder to extract text input. Text features ; based on local visual features and text features The set formed as a joint representation of features .

[0013] Preferably, in step two, the undirected graph Relationship weights between different nodes The method to obtain it is as follows:

[0014]

[0015] in, For structural similarity; For semantic similarity; To combine global visual features And the spatial relationship of the local features of the two nodes; , and This is a hyperparameter.

[0016] Preferably, in step five, the weights The method to obtain it is as follows:

[0017]

[0018] in, For feature extraction functions; It is the smallest bounding rectangle of the set of boundary points; It is the smallest positively circumscribed rectangle of the set of boundary points; It is the proportion of area; , and This is a hyperparameter.

[0019] Preferably, the minimum bounding rectangle The method for obtaining it is as follows:

[0020] The set of boundary points of the pixel mask is extracted using the convex hull algorithm, and a rotation matrix is ​​used. Rotate the set of boundary points to obtain the minimum and maximum x and y coordinates in the rotated set; use the minimum and maximum x and y coordinates as the two endpoints of the diagonal of the minimum bounding rectangle, and then adjust the values ​​according to the rotation angle. Get the The minimum bounding rectangle of the nodes .

[0021] Preferably, in step five, the cutting angle... The method to obtain it is as follows:

[0022] Obtain the rotation angle of nodes in the candidate centroid group Standard deviation If the standard deviation is less than the set threshold, then the cutting angle will be adjusted. Equal to the mean of the rotation angles of the nodes in the candidate centroid set. If the standard deviation is greater than or equal to the set threshold, then the cutting angle... The expression is:

[0023]

[0024] wherein, is the rotation angle of the zth node in the candidate gravity center group.

[0025] As a preferred, in the step five, the method of obtaining the composition mode is as follows:

[0026] Setting a composition template according to the composition mode; calculating the similarity of the instance distribution of the candidate gravity center group and different composition templates , the expression of which is:

[0027]

[0028] wherein, ; is the visual area concerned by the zth composition template; ; ; is the number of composition templates; is the node centroid position;

[0029] In all the similarities corresponding to each candidate gravity center group, the similarity greater than the preset similarity threshold is selected , and the corresponding composition mode is taken as the composition mode of the candidate gravity center group .

[0030] As a preferred, the expression of the node centroid position is as follows:

[0031]

[0032] wherein, is the pixel mask of the zth node; represents the pixel point belonging to the pixel mask .

[0033] As a preferred, in the step five, the method of obtaining the final candidate frame is as follows:

[0034] Matching the cropping ratio closest to the minimum circumscribed frame , and the expression of the cropping ratio is as follows:

[0035]

[0036] wherein, is the ratio of the minimum circumscribed frame containing all nodes in the zth candidate gravity center group;

[0037] ​​The length or width of the minimum circumscribed rectangle is amplified to obtain an initial candidate frame, and the aspect ratio of the initial candidate frame is adjusted to the cropping ratio The size and position of the initial candidate frame are adjusted according to the golden ratio and the composition method respectively to obtain a final candidate frame.

[0038] In a second aspect, the present application provides a semantic perception-based image cropping device for implementing the above-mentioned semantic perception-based image cropping method, comprising a feature extraction module, a gravity group screening module and an image cropping module; the feature extraction module is used for extracting image features and text features of an original image; the gravity group screening module is used for obtaining a candidate gravity group according to the image features and the text features; and the image cropping module confirms a final candidate frame based on the characteristics of the candidate gravity group, and crops the original image.

[0039] The present application has the following beneficial effects:

[0040] 1. The present application effectively compensates for the limitations of traditional methods that rely only on visual features. By introducing a concept dictionary to enrich semantic information, and further mining potential semantic relationships through semantic similarity, it helps to fully exploit and utilize semantic information, thereby establishing a more rich and unified concept space between images and semantic labels, and further improving the accuracy of the cropping result and the robustness of the model.

[0041] 2. The present application breaks through the limitations of traditional methods that are fixed in angle. By considering the angle information of each instance comprehensively, the direction of the cropping frame is dynamically adjusted, so that the cropping result can more flexibly and accurately adapt to the layout and structure of the image content.

[0042] 3. The present application deeply integrates photography theory and computer vision. By comprehensively considering the geometric properties, direction information, density distribution and other features of the instance, the appropriate cropping method and composition ratio are automatically selected, greatly reducing the need for manual intervention, and having strong universality and expansibility. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The flowchart of the present application embodiment 1.

[0044] Figure 2 The frame schematic diagram of the present application embodiment 1.

[0045] Figure 3 The node weighting schematic diagram in the present application embodiment 1.

[0046] Figure 4 The comparison diagram of the minimum circumscribed rectangle and the minimum positive circumscribed rectangle in the present application embodiment 1.

[0047] Figure 5 The schematic diagram of the attention area of different composition methods in the present application embodiment 1.

[0048] Figure 6 The cutting result comparison diagram of the present application and the existing image cutting method.

[0049] Figure 7 The structure diagram of embodiment 2 of the present application. DETAILED DESCRIPTION

[0050] The present application is further described below in conjunction with the drawings.

[0051] Embodiment 1

[0052] As shown in Figure 1 and Figure 2 , an image cutting method based on semantic perception includes the following steps:

[0053] Step 1, obtaining an original image ; wherein, and are the height and width of the original image respectively; is the channel number of the original image. The pixel mask and semantic label corresponding to each instance in the original image are obtained by using a segmentation detection model ; wherein, ; is the number of instances in the original image; is a set of semantic categories constructed based on a concept database such as Things; in order to ensure universality, concepts not defined in the concept dictionary WordNet will be filtered out in the set of semantic categories.

[0054] Based on the semantic label , the detailed definition of each semantic label is obtained by using a concept dictionary, and its expression is:

[0055]

[0056] wherein, is a concept dictionary.

[0057] The concept dictionary includes classic language knowledge bases such as WordNet and BabelNet, and can obtain the definition of each label based on the semantic label, provide prior knowledge for the label through a more rich text representation form, help to represent complex semantic information in the form of feature embedding, and generate a more unified and rich concept space. This enhancement process not only improves the semantic understanding ability of the model for the label, but also provides a basis for mining the semantic relationship between different categories.

[0058] The semantic label and corresponding definitions a set of components as text input .

[0059] Step two, extract global visual features of the original image by image encoder and pixel mask local visual features global visual features and local visual features are represented as:

[0060]

[0061]

[0062] wherein, is the image encoder; and is the spatial resolution of global visual features; is the dimension of global visual features.

[0063] extract text features of text input by text encoder text features text features are represented as:

[0064]

[0065] wherein, is the text encoder.

[0066] According to local visual features and text features construct the feature joint representation of each instance , which is convenient for subsequent graph structure construction and analysis.

[0067] Step three, construct a weighted undirected graph and adjacency matrix , which is represented as:

[0068]

[0069] wherein, is the node set, ; is the instance; is the edge set between nodes; is the relationship weight between the instance and the instance; .

[0070] The method of obtaining the relationship weight between different instances is as follows:

[0071] a. Obtain the undirected graph according to the brightness, contrast and structure information of the pixel mask corresponding to different instances The structural similarity between two different nodes in the undirected graph The expression is:

[0072] sim a,b struct =[ l( M a , M b ) ] α 1 [ c( M a , M b ) ] α 2 [ s( M a , M b ) ] α 3

[0073] wherein, is the brightness contrast function; is the contrast contrast function; is the structure contrast function; , and is a hyperparameter, for controlling the weight of the contribution of different modules; is the pixel mask corresponding to different instances.

[0074] b. Obtain the undirected graph according to the text input corresponding to different instances The semantic similarity between two different nodes in the undirected graph The expression is:

[0075]

[0076] wherein, is the text input corresponding to different instances.

[0077] c. Calculate the relationship weight between nodes based on the semantic similarity , structural similarity and spatial relationship between two nodes The expression is:

[0078] ​

[0079] in, , and For hyperparameters; To combine global visual features And the spatial relationship between the local features of the two nodes.

[0080] hyperparameters and Hyperparameters used to control the weights of each similarity level Weights used to control location information, global visual features Local features can be used and By guiding and adjusting the distance calculation between local features, we can better capture the spatial relationships between nodes.

[0081] Step 4: Obtain the weights corresponding to the nodes.

[0082] 4-1. For example Figure 3 and Figure 4 As shown, the centroid position of each node is obtained respectively. The smallest positively circumscribed rectangle and the smallest bounding rectangle; the centroid position of each node. The expression is:

[0083]

[0084] in, Indicates belonging to pixel mask The pixels.

[0085] The two endpoints of the diagonal of the smallest positive circumscribed rectangle are respectively denoted as and ;in, Find the smallest positive bounding rectangle based on the coordinates of the two endpoints of the diagonal. .

[0086] The method for obtaining the minimum bounding rectangle is as follows:

[0087] a. Extracting pixel masks using the convex hull algorithm set of boundary points It is represented as:

[0088]

[0089] in, Represents the convex hull algorithm; The coordinates of the k-th boundary point; For the first The number of boundary points for each instance; .

[0090] b. Define the rotation matrix ; wherein, is the rotation angle of the kth instance; the boundary point set is rotated to different angles by the rotation matrix to obtain the rotated boundary point set

[0091]

[0092]

[0093] wherein, is the coordinate of the kth rotated boundary point.

[0094] c. Obtain the rotated forward rectangular rotation boundary according to the boundary point set , which is expressed as:

[0095]

[0096] wherein, and are the minimum horizontal coordinate and vertical coordinate in the boundary point set , respectively, and are the maximum horizontal coordinate and vertical coordinate in the boundary point set , respectively; and are the horizontal and vertical coordinates of the boundary points in the boundary point set .

[0097] d. Calculate the rotated rectangular area based on the forward rectangular rotation boundary , which is expressed as:

[0098]

[0099] e. Select the rotation angle that minimizes the area of the forward rectangular rotation boundary as the rotation angle of the corresponding instance , which is expressed as:

[0100]

[0101] wherein, is the input value corresponding to the value of the minimized variable.

[0102] f. The two endpoints of the diagonal line of the minimum circumscribed rectangle are represented as and ; according to the two endpoints of the diagonal line of the minimum circumscribed rectangle and the minimum rotation angle ​​Get the Minimum bounding rectangle of each instance .

[0103] 4-2. To account for size factors, calculate the area ratio of each instance. Its expression is:

[0104]

[0105] in, pixel mask The total number of midpoints; .

[0106] 4-3. Taking into account visual attributes such as angle and size, obtain the weight corresponding to each node. Its expression is:

[0107]

[0108] in, For feature extraction functions; , and This is a hyperparameter used to control the weights of different visual attributes.

[0109] The feature extraction function normalizes and merges the area of ​​the minimum bounding rectangle or the minimum positive bounding rectangle, and then maps it to quantitative feature values ​​related to weight calculation.

[0110] Step 5: Utilize a Graph Convolutional Network (GCN) to jointly represent the features of the nodes. Modeling is performed; for each node, it is necessary to consider all its neighboring nodes and the feature information contained within itself. The way graph convolutional networks handle edges and nodes is represented as follows:

[0111]

[0112]

[0113]

[0114] in, It is the first Layer node feature matrix; The node features are obtained after graph convolution; It is a non-linear activation function; It is a degree matrix; This is the processed adjacency matrix; For the first Layer weight matrix; For hyperparameters, when When the value of is 1, it means that the importance of the node's own features is the same as the importance of its neighbors; for An identity matrix of 3D; For nodes The sum of the weights of connected edges; For nodes and nodes The edge weight between nodes, if nodes and nodes If there is no connection, the weight is 0.

[0115] Based on the adjacency matrix and degree matrix calculated by the graph convolutional network, construct the Laplacian matrix; calculate the minimum value of the Laplacian matrix. The eigenvectors corresponding to each eigenvalue are obtained, and the obtained eigenvectors are arranged into a matrix according to columns. Q min = [ q 1 , q 2 ,..., q K ] ;in, The number of candidate centroid groups; For the first The eigenvectors corresponding to small eigenvalues; Using the K-means algorithm on the matrix Cluster all rows to obtain cluster partitions, and construct a set of candidate centroid groups based on the partitioned clusters. ;in, For the first The instance information contained in each candidate centroid group.

[0116] Step 6: Crop the original image

[0117] 6-1. Obtain the cropping angle

[0118] The standard deviation of the node rotation angles in the candidate centroid set. To determine the consistency of node orientation, use the standard deviation. The expression is:

[0119]

[0120]

[0121] in, For the first The number of nodes in the candidate centroid group; Let be the rotation angle of the z-th node in the candidate centroid group; For the first the mean value of the rotation angle of the candidate gravity group; .

[0122] If the standard deviation is less than a set threshold value, it can be considered that the directions of the nodes are highly consistent, and the cutting angle of the candidate gravity group is the mean value of the rotation angle of the candidate gravity group . ; wherein the threshold value of the standard deviation is set to [10° ,20 °] .

[0123] If the standard deviation is greater than or equal to the set threshold value, it means that the directions of the nodes are greatly different, and the overall angle cannot be directly used as the direction of the cutting frame. In this case, the node with the largest weight is selected as the leading node, and the influence of other nodes on the cutting angle is determined according to the weight proportion, and the expression of the cutting angle is:

[0124]

[0125] wherein, is the weight of the zth node in the candidate gravity group.

[0126] 6-2. Obtain the cutting ratio

[0127] According to the cutting angle , the minimum circumscribed rectangle covering all nodes in the tth candidate gravity group is obtained as the minimum circumscribed frame, and the closest cutting ratio is automatically matched .

[0128]

[0129] wherein, is the ratio of the minimum circumscribed frame containing all nodes in the tth candidate gravity group.

[0130] 6-3. Obtain the composition method

[0131] As shown in Figure 5 , according to the composition method, a series of composition templates are set up. Each composition template focuses on different visual areas. For example, in the horizontal line, vertical line and diagonal line composition method, the visual gravity is distributed around different auxiliary lines; in the center composition method, the visual gravity is concentrated in the center position to present a balanced and symmetrical beauty; in the three-part composition method and the golden ratio composition method, the visual gravity is distributed in the intersection area of the auxiliary lines.

[0132] According to the distribution of each candidate gravity group, the similarity of each candidate gravity group with different composition templates is calculated , and the expression is: ​

[0133]

[0134] wherein, ; is the visual area of interest for the th composition template; ; is the number of composition templates; is the centroid position of the th node in the candidate center group.

[0135] Among all the similarity corresponding to each candidate center group, the similarity greater than the preset similarity threshold is selected as the composition mode of the candidate center group.

[0136] 6-4. Enlarge the length or width of the minimum bounding box to obtain an initial candidate box, and make the aspect ratio of the initial candidate box the same as the cropping ratio Adjust the size (content compactness ) and position of the initial candidate box according to the golden ratio and the composition mode respectively to obtain a final candidate box, and the adjustment method is as follows:

[0137]

[0138] wherein, and are the width and height of the final candidate box respectively; and are the width and height of the initial candidate box respectively; ; is the number of composition modes .

[0139] Crop the original image using the final candidate box to obtain a cropping result.

[0140] Step seven, method evaluation

[0141] For example Figure 6As shown, compared with the cropping results of the existing image cropping method, the existing image cropping method ignores the direction information of instances in the image, adopts a fixed cropping frame angle, ignores the direction difference between the cropping results, and leads to a lack of flexibility in the cropping results. The present application calculates the minimum circumscribed rectangle and the minimum positive circumscribed rectangle of each instance, and introduces an angle factor for calculation. When the rotation angles in the candidate barycenter group are highly consistent, the overall angle is directly used as the direction of the cropping frame, and in the case of large angle difference, the cropping frame angle is adjusted according to the node with the largest weight, so as to ensure that the cropping frame is more consistent with the layout and structure of the actual content. At the same time, the present application optimizes the image cropping and composition process by combining the aesthetic principles in photography theory. When processing the candidate barycenter group, the system selects appropriate composition methods such as centralized composition or symmetrical composition according to the centroid distribution density and geometric layout to ensure visual focus and overall harmony. The present application also considers the content compactness, and the system can intelligently adjust the cropping method to ensure the balance and tension of the image. In addition, the system also dynamically adjusts the aspect ratio according to the size of the image and the proportion rules in photography theory to ensure the best visual effect. Through this combination of aesthetics and computer vision technology, the present application realizes the organic integration of artistry and technology, which helps to improve the aesthetic value of the cropping results.

[0142] Embodiment 2

[0143] As Figure 7 shown, an image cropping device based on semantic perception is used to implement the cropping method in embodiment 1, which includes a feature extraction module, a barycenter group screening module and an image cropping module; the feature extraction module is used to extract image features and text features of the original image; the barycenter group screening module is used to obtain candidate barycenter groups according to the image features and the text features; the image cropping module confirms the final candidate frame based on the characteristics of the candidate barycenter groups, such as the standard deviation of the rotation angle of the candidate barycenter group, the weight of the elements in the group, the size of the bounding box, and the content compactness, and crops the original image.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limiting; obviously, the drawings are only some examples or embodiments of the present application, and those skilled in the art can also apply the present application to other similar situations according to these drawings without creative labor. In addition, it can be understood that although the work done in this development process may be complex and long, but for ordinary skilled in the art, some design, manufacture or production changes according to the technical content disclosed in the present application are only routine technical means, and should not be regarded as the content disclosed in the present application is insufficient.

[0145] Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood that modifications can be made to the foregoing embodiments, or additional implementations can be implemented, without departing from the spirit and scope of the inventive subject matter. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The protection scope of the present application is defined by the appended claims.

Claims

1. A semantically aware image cropping method, characterized in that: Includes the following steps: Step 1: Obtain the original image ; Obtain the original image using a segmentation and detection model The pixel mask corresponding to each instance and semantic tags According to pixel mask and semantic tags Obtaining joint feature representation ;in, ; Original image The number of instances in; Step 2: Using the original image Use instances as nodes to construct a weighted undirected graph. and adjacency matrix ; Step 3: Extract pixel mask set of boundary points ; through rotation matrix Set of boundary points Rotate to different angles to obtain the rotation angle. Rotation angle This minimizes the area of ​​the positively rotated rectangle's boundary. Step 4: Use a graph convolutional network to jointly represent features and weighted undirected graph Construct the Laplacian matrix; use spectral clustering to segment the eigenvalues ​​and eigenvectors of the Laplacian matrix, and use each cluster obtained from the segmentation as a group of candidate centroids; Step 5: Based on the weights of the nodes in each candidate centroid group and rotation angle Determine the corresponding cutting angle ;in, ; For the first The number of nodes in the candidate centroid group; ; The number of candidate centroid groups; based on the cropping angle. Get coverage of the The smallest bounding rectangle of all nodes in the candidate centroid group is used as the smallest bounding box, and the final candidate boxes are obtained through a mapping method; the final candidate boxes are then used to refine the original image. Perform cropping to obtain the cropping result.

2. The image cropping method based on semantic awareness according to claim 1, characterized in that: In step one, the joint representation of features The method to obtain it is as follows: Obtain each semantic tag using a concept dictionary Detailed definition ; using semantic tags and the corresponding definition The set formed as text input Use image encoder pixel mask Local visual features ; Extract text input using a text encoder. Text features ; based on local visual features and text features The set formed as a joint representation of features .

3. The image cropping method based on semantic awareness according to claim 1, characterized in that: In step two, the undirected graph Relationship weights between different nodes The method to obtain it is as follows: ； in, For structural similarity; For semantic similarity; To combine global visual features And the spatial relationship of the local features of the two nodes; , and This is a hyperparameter.

4. The image cropping method based on semantic awareness according to claim 1, characterized in that: In step five, the weights The method to obtain it is as follows: ； in, For feature extraction functions; It is the minimum bounding rectangle of the set of boundary points; It is the smallest positively circumscribed rectangle of the set of boundary points; It is the proportion of area; , and This is a hyperparameter.

5. The image cropping method based on semantic awareness according to claim 4, characterized in that: The minimum bounding rectangle The method for obtaining it is as follows: The set of boundary points of the pixel mask is extracted using the convex hull algorithm, and a rotation matrix is ​​used. Rotate the set of boundary points to obtain the minimum and maximum x and y coordinates in the rotated set; use the minimum and maximum x and y coordinates as the two endpoints of the diagonal of the minimum bounding rectangle, and then adjust the values ​​according to the rotation angle. Get the The minimum bounding rectangle of the nodes .

6. The image cropping method based on semantic awareness according to claim 1, characterized in that: In step five, the cutting angle The method to obtain it is as follows: Obtain the rotation angle of nodes in the candidate centroid group Standard deviation ; If the standard deviation is less than the set threshold, then the cutting angle... Equal to the mean of the rotation angles of the nodes in the candidate centroid set. If the standard deviation is greater than or equal to the set threshold, then the cutting angle... The expression is: ； in, Let be the rotation angle of the z-th node in the candidate centroid group.

7. The image cropping method based on semantic awareness according to claim 1, characterized in that: In step five, the method for obtaining the composition method is as follows: Establish composition templates based on composition methods; calculate the similarity between the instance distribution of candidate centroid groups and different composition templates. Its expression is: ； in, ; For the first The visual area that a composition template focuses on; ; The number of composition templates; The location of the centroid of the node; All similarities corresponding to each candidate centroid group In the selection, similarity values ​​greater than a preset similarity threshold are chosen. The corresponding composition method, serving as the composition method for this candidate center group. .

8. The image cropping method based on semantic awareness according to claim 7, characterized in that: The location of the centroid of the node The expression is: ； in, Let z be the pixel mask of the z-th node; Indicates belonging to pixel mask The pixels.

9. The image cropping method based on semantic awareness according to claim 1, characterized in that: In step five, the method for obtaining the final candidate box is as follows: Match the clipping ratio closest to the minimum bounding box. Cutting proportions The expression is: ； in, For including the first The proportion of the minimum bounding box of all nodes within the candidate centroid group; Enlarge the length or width of the smallest bounding rectangle to obtain the initial candidate box, and adjust the aspect ratio of the initial candidate box to match the clipping ratio. The same applies; adjust the size and position of the initial candidate boxes according to the golden ratio and composition method respectively to obtain the final candidate boxes.

10. An image cropping device based on semantic awareness, characterized in that: The method for implementing the semantically aware image cropping method of claim 1 includes a feature extraction module, a centroid group filtering module, and an image cropping module. The feature extraction module is used to extract image features and text features of the original image. The centroid group filtering module is used to obtain candidate centroid groups based on the image features and text features. The image cropping module confirms the final candidate box based on the characteristics of the candidate centroid groups and crops the original image.

Images