Image cropping method, model training method, device, electronic equipment and medium

Abstract

Embodiments of the present disclosure disclose an image cropping method, a model training method, an apparatus, an electronic device and a medium, wherein the image cropping method comprises: segmenting a to-be-cropped image to obtain a first segmentation image, determining a bounding box of a target object in the to-be-cropped image according to the first segmentation image; generating a first candidate box in the bounding box, selecting a first target box from the first candidate box according to an aesthetic score of a first feature map corresponding to the first candidate box; and taking an image in the first target box in the to-be-cropped image as a cropping result. By determining the bounding box of the target object in the to-be-cropped image based on the segmentation image and generating the first candidate box in the bounding box, the cropping interval can be reduced, the number of candidate boxes generated can be greatly reduced, the time consumption of the scoring process can be reduced, and the real-time performance of cropping can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image cropping method, a model training method, an apparatus, an electronic device, and a medium. Background Technology

[0002] In existing image cropping algorithms that employ aesthetic evaluation, a large number of candidate boxes are typically generated on the global image. The image features corresponding to these candidate boxes are then fed into a scorer for aesthetic evaluation, and the image is cropped based on the candidate box with the highest score. The shortcomings of existing technologies include at least the following: the excessive number of candidate boxes leads to a lengthy scoring process, resulting in poor real-time cropping performance. Summary of the Invention

[0003] This disclosure provides an image cropping method, a model training method, an apparatus, an electronic device, and a medium that can improve the real-time performance of cropping.

[0004] In a first aspect, embodiments of this disclosure provide an image cropping method, including:

[0005] The image to be cropped is segmented to obtain a first segmented image, and the bounding box of the target object in the image to be cropped is determined based on the first segmented image.

[0006] A first candidate box is generated within the bounding box, and a first target box is selected from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box.

[0007] The image located within the first target box in the image to be cropped is taken as the cropping result.

[0008] Secondly, embodiments of this disclosure provide a model training method, including:

[0009] Obtain the first sample image, the segmentation label of the first sample image, and the aesthetic rating label corresponding to the sample cropping box of the first sample image;

[0010] The first sample image is used to extract features to obtain a third feature map;

[0011] The third feature map is reconstructed using a segmentation model to obtain a second segmented image, and the segmentation model is trained based on the second segmented image and the segmentation label.

[0012] A second candidate box is generated within the first sample image, and a fourth feature map corresponding to the second candidate box is determined based on the third feature map and the second candidate box.

[0013] The aesthetic scoring model outputs the predicted score of the fourth feature map, and the aesthetic scoring model is trained based on the predicted score and the aesthetic score label.

[0014] The trained segmentation model is used to determine the first segmented image in any of the image cropping methods described in this disclosure; the trained aesthetic scoring model is used to determine the aesthetic score in any of the image cropping methods described in this disclosure.

[0015] Thirdly, embodiments of this disclosure also provide an image cropping apparatus, comprising:

[0016] The bounding box determination module is used to segment the image to be cropped to obtain a first segmented image, and to determine the bounding box of the target object in the image to be cropped based on the first segmented image.

[0017] The target box determination module is used to generate a first candidate box within the bounding box, and select a first target box from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box;

[0018] The cropping module is used to take the image located within the first target box in the image to be cropped as the cropping result.

[0019] Fourthly, embodiments of this disclosure also provide a model training apparatus, comprising:

[0020] The sample acquisition module is used to acquire a first sample image, the segmentation label of the first sample image, and the aesthetic rating label corresponding to the sample cropping box of the first sample image.

[0021] The feature extraction module is used to extract features from the first sample image to obtain a third feature map;

[0022] The segmentation model training module is used to reconstruct the features of the third feature map using the segmentation model to obtain a second segmentation image, and to train the segmentation model based on the second segmentation image and the segmentation label;

[0023] The candidate box feature determination module is used to generate a second candidate box in the first sample image and determine a fourth feature map corresponding to the second candidate box based on the third feature map and the second candidate box.

[0024] An aesthetic scoring model training module is used to output the predicted score of the fourth feature map through the aesthetic scoring model, and to train the aesthetic scoring model based on the predicted score and the aesthetic scoring label.

[0025] The trained segmentation model is used to determine the first segmented image in any of the image cropping methods described in this disclosure; the trained aesthetic scoring model is used to determine the aesthetic score in any of the image cropping methods described in this disclosure.

[0026] Fifthly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0027] One or more processors;

[0028] Storage device for storing one or more programs.

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the image cropping method as described in any embodiment of the present disclosure, or implement the model training method as described in any embodiment of the present disclosure.

[0030] Sixthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image cropping method as described in any embodiment of this disclosure, or to implement a model training method as described in any embodiment of this disclosure.

[0031] The technical solution of this disclosure includes an image cropping method comprising: segmenting an image to be cropped to obtain a first segmented image; determining the bounding box of a target object in the image to be cropped based on the first segmented image; generating a first candidate box within the bounding box; selecting a first target box from the first candidate box based on the aesthetic score of a first feature map corresponding to the first candidate box; and using the image located within the first target box in the image to be cropped as the cropping result. By determining the bounding box of the target object in the image to be segmented based on the segmented image and generating a first candidate box within the bounding box, the cropping range can be narrowed, the number of candidate boxes generated can be greatly reduced, thereby reducing the time consumed in the scoring process and improving the real-time performance of cropping. Attached Figure Description

[0032] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0033] Figure 1 This is a schematic flowchart of an image cropping method provided in Embodiment 1 of this disclosure;

[0034] Figure 2 This is a flowchart of an image cropping method provided in Embodiment 1 of this disclosure;

[0035] Figure 3 This is a flowchart of an image cropping method provided in Embodiment 2 of this disclosure;

[0036] Figure 4 This is an example image of the cropping result corresponding to different cropping ratios in an image cropping method provided in Embodiment 3 of this disclosure;

[0037] Figure 5 This is a flowchart of an image cropping method provided in Embodiment 3 of this disclosure;

[0038] Figure 6 This is a schematic flowchart of a model training method provided in Embodiment 4 of this disclosure;

[0039] Figure 7 This is a flowchart of a model training method provided in Embodiment 4 of this disclosure;

[0040] Figure 8 This is a schematic diagram of the structure of an image cropping device provided in Embodiment 5 of this disclosure;

[0041] Figure 9 This is a schematic diagram of the structure of a model training device provided in Embodiment Six of this disclosure;

[0042] Figure 10 This is a schematic diagram of the structure of an electronic device provided in Embodiment 7 of this disclosure. Detailed Implementation

[0043] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0044] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0045] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0046] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0047] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0048] Example 1

[0049] Figure 1 This is a flowchart illustrating an image cropping method according to Embodiment 1 of this disclosure. This embodiment is applicable to image cropping, and particularly to cropping images containing salient objects. The method can be executed by an image cropping device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a mobile phone or computer.

[0050] like Figure 1 As shown, the image cropping method provided in this embodiment may include:

[0051] S110. The image to be cropped is segmented to obtain a first segmented image, and the bounding box of the target object in the image to be cropped is determined based on the first segmented image.

[0052] In this embodiment, the image to be cropped can be a currently acquired image or a target image read from a preset storage space, and the image to be cropped can be an image of any resolution; the first segmented image can be considered as the image after semantic segmentation of the image to be cropped. Semantic segmentation refers to pixel-by-pixel classification prediction based on semantics as the dividing criterion, where each semantic category can represent different individual objects or the same type of object. In the first segmented image, pixels belonging to different semantic categories can be distinguished using different formats, such as different colors or different grayscale levels.

[0053] This process involves semantic segmentation of the image to be cropped using an image semantic segmentation algorithm to obtain a first segmented image. The image semantic segmentation algorithm may include, but is not limited to, traditional semantic segmentation algorithms based on random forest classifiers, and network model segmentation algorithms based on deep learning.

[0054] The system can present the obtained first segmented image and prompt the user to select the desired semantic category for cropping; then, the object selected by the user in the semantic category can be used as the target object. Alternatively, a saliency analysis can be performed on the obtained first segmented image, and objects in the semantic categories with saliency can be used as the target objects. Specifically, the saliency analysis can be performed based on the proportion of pixels belonging to the same semantic category out of all pixels; or, it can be based on the proportion of the area of ​​the connected regions of pixels belonging to the same semantic category in the first segmented image, etc.

[0055] Typically, the image to be cropped and the first segmented image have the same resolution. When the accuracy of the first segmented image is high, the image regions representing the same object in the image to be cropped and the corresponding first segmented image are the same. Therefore, the bounding box of the target object in the first segmented image can be determined based on the set of pixels corresponding to the target object in the first segmented image, and this bounding box can then be used as the bounding box of the target object in the image to be cropped.

[0056] The bounding box can be a closed box that includes the entire target object and is located at a distance greater than a first preset value and less than a second preset value from the target object's outline. Furthermore, the bounding box can be, for example, a rectangle or an irregular polygonal box adaptively generated based on the shape of the target object. Preferably, the bounding box is a rectangle, which facilitates the generation of candidate boxes with a certain clipping ratio within the bounding box.

[0057] S120. Generate a first candidate box within the bounding box, and select a first target box from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box.

[0058] In this embodiment, the first candidate box can represent a candidate cropping range determined for image cropping, and the first target box can represent the final cropping range determined for image cropping. Multiple first candidate boxes can be generated in a sliding window manner within the bounding box of the image to be cropped, and features are extracted from the image within each first candidate box to obtain a first feature map. Each first feature map can be input into a pre-trained aesthetic scoring model, so that the aesthetic scoring model outputs an aesthetic score for each first feature map.

[0059] In this approach, the first candidate bounding box corresponding to the highest aesthetic score can be used as the first target bounding box. Alternatively, the first candidate bounding boxes corresponding to the top N aesthetic scores can be displayed, and the user can be prompted to select the desired cropping range; then, the first candidate bounding box corresponding to the cropping range selected by the user can be used as the first target bounding box, where N can be an integer greater than or equal to 1.

[0060] Compared to traditional image cropping schemes that generate candidate boxes globally, this embodiment first determines the bounding box of the target object and then generates candidate boxes within that bounding box. This significantly reduces the number of candidate boxes, experimentally verified to be 10-20 times less. This not only reduces the time and storage space required for extracting features corresponding to the candidate boxes but also decreases the time spent by the aesthetic scoring model, thereby improving the real-time performance of image cropping. Furthermore, when the bounding box is determined based on the salient objects in the image to be cropped, it enhances the aesthetic scoring model's preference for salient objects. Generating candidate boxes within the bounding box also avoids cropping the wrong object location.

[0061] S130. The image located within the first target box in the image to be cropped is taken as the cropping result.

[0062] After determining the first target bounding box, the image to be cropped can be cropped according to the first target bounding box, and the image within the first target bounding box in the image to be cropped can be retained as the cropping result.

[0063] For example, Figure 2 This is a flowchart illustrating an image cropping method provided in Embodiment 1 of this disclosure. See also... Figure 2 First, a first segmented image can be obtained by performing salient segmentation on the image to be cropped. Second, the bounding box of the target object in the first segmented image can be determined and used as the bounding box of the target object in the image to be cropped. Third, multiple first candidate boxes can be generated inside the bounding box of the image to be cropped, and features can be extracted from the images within each first candidate box to obtain each first feature map. Next, the aesthetic score of each first feature map can be determined, and the first target box can be determined from the first candidate boxes based on each aesthetic score. Finally, the image in the image to be cropped that is located within the first target box is used as the cropping result.

[0064] This embodiment can be applied to situations with high real-time requirements and / or limited resources, such as image cropping on mobile devices with limited computing / storage resources. By determining the bounding box of the target object based on the segmented image, the cropping range can be significantly narrowed, thereby reducing the number of candidate boxes generated, saving computation and storage, and improving the real-time performance of cropping on mobile devices. Furthermore, determining the final target box through aesthetic scoring ensures that the cropping result is aesthetically pleasing, guaranteeing cropping effectiveness.

[0065] The technical solution of this disclosure involves segmenting an image to be cropped to obtain a first segmented image, determining the bounding box of the target object in the image to be cropped based on the first segmented image, generating a first candidate box within the bounding box, selecting a first target box from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box, and using the image within the first target box in the image to be cropped as the cropping result. By determining the bounding box of the target object in the image to be segmented based on the segmented image and generating a first candidate box within the bounding box, the cropping range can be narrowed, the number of candidate boxes generated can be greatly reduced, thereby reducing the time consumption of the scoring process and improving the real-time performance of cropping.

[0066] Example 2

[0067] This embodiment can be combined with various optional schemes in the image cropping method provided in the above embodiments. The image cropping method provided in this embodiment describes in detail the steps of determining the first segmented image, determining the bounding box, and generating the first candidate box.

[0068] The segmentation model allows for feature reconstruction to obtain the first segmentation image. Since the image to be cropped and the first segmented image have the same resolution, the bounding box of the target object in the image to be cropped can be determined using the position coordinates of the pixel set corresponding to the target object in the first segmentation image. Furthermore, first candidate boxes can be generated based on the cropping ratio input by the user, and / or a corresponding number of first candidate boxes can be generated based on the cropping precision used as input, thus enabling flexible generation of candidate boxes.

[0069] In some optional implementations, segmenting the image to be cropped to obtain a first segmented image may include: extracting features from the image to be cropped to obtain a second feature map, and reconstructing the second feature map using a segmentation model to obtain the first segmented image; correspondingly, the first feature map is determined based on the second feature map and the first candidate box.

[0070] This process involves multi-level downsampling of the image to be cropped to extract feature maps at different levels, with each level's feature map potentially belonging to the second feature map category. Higher-level feature maps have lower resolution and possess more semantic information but lack spatial information; conversely, lower-level feature maps have higher resolution and possess more refined spatial information but lack semantic information. Spatial information can be understood as the relative spatial positions or orientations of multiple objects in the image, while semantic information can be understood as the semantic attributes of the objects contained in the image.

[0071] For example, Figure 3 This is a flowchart illustrating an image cropping method provided in Embodiment 2 of this disclosure. See also... Figure 3The image to be cropped can be downsampled through multiple layers of network layers 1-8 to extract feature maps at different levels. Among them, network layer 1 can be a network layer containing convolutional layers (Conv), batch normalization (BN) layers, and rectified linear unit (ReLU) activation layers; network layers 2-8 can all be inverted residual layers proposed by MobileNetV2.

[0072] Specifically, the feature map output by network layer 3 can be half the resolution of the image to be cropped, the feature map output by network layer 4 can be one-quarter the resolution of the image to be cropped, the feature map output by network layer 6 can be one-eighth the resolution of the image to be cropped, and the feature map output by network layer 8 can be one-sixteenth the resolution of the image to be cropped. It can be considered that the feature map output by network layer 3 is a lower-level feature map, and the feature map output by network layer 8 is a higher-level feature map.

[0073] See you again Figure 3 The feature map output from network layer 8 can be reconstructed through network layers 14-16 to restore the high-level feature map to its original resolution and achieve pixel-by-pixel semantic attribute classification, thus obtaining the first segmentation map. The segmentation model can be composed of network layers 14-16, and any layer in network layers 14-16 can be composed of Conv, BN, and ReLU. The segmentation model can adopt a U-net structure.

[0074] Figure 3 In the process of restoring high-level feature maps to their original resolution, feature maps of the same resolution from the downsampling process can be skipped after upsampling of the current-level feature map. This allows for the supplementation of spatial information on top of semantic information, thus achieving feature fusion. For example, Figure 3 The feature map output by the middle network layer 14 is upsampled by 2 times (represented by "×2" in the figure) and then fused with the feature map output by the network layer 6 (represented by the circled letter C in the figure).

[0075] The feature map output by network layer 8 can be upsampled by 2 times, and then concatenated with the feature map output by network layer 6 and the feature map output by network layer 4 after being downsampled by 2 times (represented by " / 2" in the figure). The concatenated image is then convolved by network layer 9 (e.g., Conv layer) to obtain the final feature map, which is also a second feature map.

[0076] After determining the first segmented image, the bounding boxes of the target objects in the image to be cropped can be determined, thereby generating first candidate boxes within the bounding boxes of the image to be cropped. At this point, the features within the range corresponding to the first candidate boxes in the final feature map can be used as the first feature maps corresponding to each first candidate box. Since the final feature map and the image to be cropped have a correspondence based on resolution compression, the range of the first candidate box mapped to the final feature map can be determined according to this correspondence, and the features within the mapped range can then constitute the first feature map corresponding to the first candidate box.

[0077] In these optional implementations, the first segmented image can be generated using a segmentation model, such as a saliency segmentation branch network. Furthermore, an aesthetic scoring model can be composed of network layers 10-13, where any layer in network layers 10-11 can be composed of Conv, BN, and ReLU, and network layers 12-13 can be fully connected layers. After obtaining the first feature map, each first feature map can be processed through network layers 10-13 to determine its aesthetic score (score in the figure). Then, a first target box can be selected from the first candidate boxes based on the aesthetic score, and the image within the first target box in the image to be cropped is taken as the cropping result.

[0078] In some optional implementations, determining the bounding box of the target object in the image to be cropped based on the first segmented image may include: determining the bounding box of the target object in the image to be cropped based on the position coordinates of the pixels in the first segmented image that belong to the semantic classification of the target object.

[0079] Since the bounding boxes of the same object can have the same position coordinates in the first segmented image and the image to be cropped, the bounding box can be determined based on the first segmented image. See also Figure 3 The bounding box can be a rectangle, and the rectangle can be represented by the coordinates of its top-left and bottom-right corners, or its top-right and bottom-left corners.

[0080] The process of determining the bounding box of the target object in the first segmented image is as follows: First, the position coordinates of each pixel in the first segmented image that belongs to the semantic category of the target object can be determined; then, the top / bottom / left / right extreme pixels can be determined, and an initial rectangular box surrounding the target object can be determined based on the position coordinates of the extreme pixels; finally, the bounding box of the target object can be obtained by expanding a certain area outward from the initial rectangular box. The position coordinates can refer to pixel coordinates.

[0081] Among these optional implementations, the initial rectangle can be determined by the position coordinates of the target object's pixels, and the bounding box can be expanded based on the initial rectangle. This can help the target object occupy a more suitable area and position in the cropping result, thereby ensuring the cropping effect.

[0082] In some optional implementations, generating a first candidate box within the bounding box may include: generating a first candidate box that conforms to the input clipping ratio within the bounding box; and / or, generating a number of first candidate boxes within the bounding box corresponding to the input clipping precision.

[0083] The cropping ratio can be any aspect ratio of the image input by the user, such as 4:3, 3:4, 1:1, 9:16, or 16:9. Multiple first candidate boxes of varying sizes but the same cropping ratio can be generated by sliding windows of different sizes within the bounding box. For example, Figure 4 These are example images showing cropping results corresponding to different cropping ratios in an image cropping method provided in Embodiment 3 of this disclosure. See also... Figure 4 The image cropping method based on the embodiments of this disclosure can crop the image to be cropped into an aspect ratio of 4:3, 1:1, 9:16 or 16:9.

[0084] The cropping precision can be a predefined precision level, such as low, medium, and high. The number of first candidate boxes corresponding to different cropping precision levels can increase progressively, and the number of first candidate boxes corresponding to different cropping precision levels can be preset. Accordingly, the desired cropping precision input by the user can be used to determine the corresponding number of first candidate boxes, and that number can be generated within the bounding box.

[0085] The first candidate bounding box can be determined based on the cropping ratio and / or cropping precision input by the user. When the user only inputs the cropping ratio, the cropping precision can be set to a default value, such as medium precision. When the user only inputs the cropping precision, the cropping ratio can be set to a default value, an optimal value, or any of the output ratios. The default value can be any of the output ratios, such as 1:1; the optimal value can be the ratio closest to the original image to be cropped. Generating the first candidate bounding box based on the optimal cropping ratio avoids cropping too much area and ensures the resolution of the cropping result. Generating the first candidate bounding box based on all output ratios provides users with richer cropping results to meet their needs.

[0086] Among these optional implementations, a first candidate box can be generated based on the cropping ratio input by the user, and / or a corresponding number of first candidate boxes can be generated based on the cropping precision used for input, thereby achieving flexible generation of candidate boxes.

[0087] The technical solution of this disclosure describes in detail the steps of determining the first segmented image, determining the bounding box, and generating the first candidate box. A first segmented image can be obtained through feature reconstruction using a segmentation model. Since the image to be cropped and the first segmented image have the same resolution, the bounding box of the target object in the image to be cropped can be determined using the position coordinates of the pixel set corresponding to the target object in the first segmented image. Furthermore, the first candidate box can be generated according to the cropping ratio input by the user, and / or a corresponding number of first candidate boxes can be generated according to the cropping precision used as input, thereby achieving flexible generation of candidate boxes.

[0088] The image cropping method provided in this embodiment belongs to the same concept as the image cropping method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0089] Example 3

[0090] This embodiment can be combined with various optional schemes in the image cropping method provided in the above embodiments. The image cropping method provided in this embodiment details the steps for determining the aesthetic score of the first feature map.

[0091] Since the number of first candidate boxes can be changed based on the cropping precision input by the user, the number of first feature maps can also vary with the cropping precision. However, the aesthetic scoring model can only process a fixed number of first feature maps. If the generated first feature maps are directly input into the aesthetic scoring model, it is easy to cause scoring anomalies, that is, it cannot score first feature maps beyond the fixed number.

[0092] In some optional implementations, after generating the first candidate box, the following steps may be included: inputting the first feature map corresponding to the first candidate box into the aesthetic scoring model in batches according to the single processing volume of the aesthetic scoring model, so that the aesthetic scoring model outputs the aesthetic score of the first feature map.

[0093] In these alternative implementations, the processing capacity of the aesthetic scoring model in a single run can be considered as the number of channels of the first feature map that can be processed at one time, which is usually set to a fixed value. When the number of first feature maps corresponding to the first candidate boxes is variable, the generated first feature maps can be input into the aesthetic scoring model in batches to perform aesthetic scoring in batches.

[0094] For example, Figure 5 This is a flowchart illustrating an image cropping method provided in Embodiment 3 of this disclosure. See also... Figure 5 In this embodiment, image cropping is achieved based on a two-part model. The first part of the model may include a segmentation model, which can be used to generate each first feature map based on the image to be cropped. The second part of the model may include an aesthetic scoring model, which can be used to receive each first feature map in batches and perform aesthetic scoring on each batch of first feature maps. Thus, it can support successful aesthetic scoring of each first feature map even when the first feature map changes.

[0095] Furthermore, with a fixed number of first feature maps corresponding to the first candidate boxes, the processing volume of the aesthetic evaluation model per run can be set to this fixed value. In this case, it is not necessary to split the model into two parts; instead, the first feature map output by the segmentation model can be directly input into the aesthetic scoring model to complete the aesthetic scoring in one go.

[0096] In some optional implementations, before inputting the first feature map corresponding to the first candidate box into the aesthetic scoring model in batches, it may also include: adjusting the first feature map corresponding to the first candidate box to a preset size.

[0097] Among these optional implementations, since the size of the first candidate box may be different when the proportions are the same, the size of the first feature map can be adjusted to a uniform preset size before inputting the first feature map into the aesthetic scoring model, which is beneficial for aesthetic scoring under a uniform standard.

[0098] The size can be adjusted using OpenCV's `resize` operation, or using C's `ROIAlign` operation, and other more complex operations. Furthermore, other preprocessing operations on the first feature map can also be applied, which will not be exhaustive here. The preset size can be set according to the actual scenario; for example, when the cropping ratio is 1:1, the preset size can be set to 9×9.

[0099] The technical solution of this disclosure describes in detail the steps for determining the aesthetic score of the first feature map. Image cropping is achieved based on a two-part model. The first part of the model may include a segmentation model, which can be used to generate each first feature map based on the image to be cropped. The second part of the model may include an aesthetic scoring model, which can be used to receive each first feature map in batches to perform aesthetic scoring on each batch of first feature maps. This allows for successful aesthetic scoring of each first feature map even when the first feature map changes.

[0100] Furthermore, the image cropping method provided in this embodiment belongs to the same disclosed concept as the image cropping method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0101] Example 4

[0102] Figure 6 This is a schematic flowchart illustrating a model training method provided in Embodiment 4 of this disclosure. This embodiment is applicable to training an image cropping model that includes a segmentation model and an aesthetic scoring model. The method can be executed by a model training device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer.

[0103] like Figure 6 As shown, the model training method provided in this embodiment may include:

[0104] S610. Obtain the first sample image, the segmentation label of the first sample image, and the aesthetic rating label corresponding to the sample cropping box of the first sample image.

[0105] In this embodiment, the first sample image can be an image obtained from an open-source database, a captured image, or an image obtained through virtual rendering, etc. The segmentation label of the first sample image can be considered as a segmented image of the first sample image. The first sample image can be labeled with multiple sample cropping boxes, and each sample cropping box can be labeled with an aesthetic rating label.

[0106] S620. Extract features from the first sample image to obtain the third feature map.

[0107] The step of extracting features from the first sample image can be referenced from the step of extracting features from the image to be cropped. The feature maps at each level corresponding to the first sample image can be referred to as the third feature maps.

[0108] S630. The second segmented image is obtained by reconstructing the features of the third feature map through the segmentation model. The segmentation model is trained based on the second segmented image and the segmentation label.

[0109] The step of reconstructing the third feature map into the second segmented image using the segmentation model can refer to the step of reconstructing the second feature map into the first segmented image using the segmentation model.

[0110] The segmentation model can be trained based on the first loss value between the second segmented image output by the segmentation model and the segmentation label. The first loss value can be calculated based on a first loss function, which can be, for example, the cross-entropy loss function (CE Loss).

[0111] S640. Generate a second candidate box within the first sample image. Based on the third feature map and the second candidate box, determine the fourth feature map corresponding to the second candidate box.

[0112] The step of generating the second candidate box within the first sample image can refer to the step of generating the first candidate box within the bounding box. The step of determining the fourth feature map corresponding to the second candidate box based on the third feature map and the second candidate box can refer to the step of determining the first feature map corresponding to the first candidate box based on the second feature map and the first candidate box.

[0113] S650. Output the predicted score of the fourth feature map through the aesthetic scoring model, and train the aesthetic scoring model based on the predicted score and aesthetic score label.

[0114] During the training of the aesthetic scoring model, the aesthetic scores corresponding to candidate boxes of different positions and sizes can be regressed based on the predicted scores of the fourth feature maps corresponding to each second candidate box output by the aesthetic scoring model. Then, the aesthetic evaluation model can be trained based on the second loss value between the regression scores corresponding to each sample cropping box and the aesthetic score labels corresponding to each sample cropping box in the regression results. The second loss value can be calculated based on a second loss function, which can be, for example, a pixel-level smooth absolute value loss function (Smooth L1 Loss).

[0115] The first and second loss functions described above are merely illustrative examples; other commonly used loss functions can also be applied. The entire network, including the segmentation model and the aesthetic scoring model, can be trained simultaneously based on the sum of the first and second loss values. Alternatively, the segmentation model can be trained separately using the first loss value, and the aesthetic scoring model can be trained using the second loss value. When both models are trained simultaneously, training is considered complete when the sum of the loss values ​​is less than a first threshold. When both models are trained separately, training is considered complete when the first loss value is less than a second threshold, and complete when the second loss value is less than a third threshold.

[0116] The trained segmentation model can be used to determine the first segmented image in any image cropping method of this disclosure embodiment; the trained aesthetic scoring model is used to determine the aesthetic score in any image cropping method of this disclosure embodiment.

[0117] For example, Figure 7 This is a flowchart illustrating a model training method provided in Embodiment 4 of this disclosure. See also... Figure 7 The first sample image can be used to extract features to obtain a third feature map; the third feature map is input into the segmentation model, and the fourth feature map determined based on the third feature map and the second candidate box is input into the aesthetic scoring model; the first loss value between the second segmentation map output by the segmentation model and the segmentation label (e.g., CE Loss in the figure) and the second loss value between the predicted score output by the aesthetic scoring model and the aesthetic score label (e.g., Smooth L1 Loss in the figure) are determined; the network containing the segmentation model and the aesthetic scoring model is trained based on the sum of the first loss value and the second loss value.

[0118] In some optional implementations, if the first sample image and aesthetic rating label belong to an aesthetic evaluation dataset, the segmentation label is obtained by segmenting the first sample image based on a pre-defined model. Common aesthetic evaluation datasets, such as the Grid Anchor based Image Cropping Dataset (GAICD), typically do not include segmentation labels for the first sample image. To obtain the segmentation label for the first sample image in the aesthetic evaluation dataset, it can be segmented based on an existing, relatively mature pre-defined model (such as the BAS-Net model), thus enabling the segmentation model to be trained based on the segmentation label.

[0119] Correspondingly, after the segmentation model and aesthetic scoring model have been trained, the process may also include: acquiring a second sample image and labeling the second sample image with segmentation labels; fixing the parameters of the aesthetic scoring model, determining a third segmentation image of the second sample image using the trained segmentation model, and optimizing the segmentation model based on the segmentation labels of the third segmentation image and the second sample image.

[0120] Because the aesthetic evaluation dataset has limited sample data, the training level of the segmentation model is relatively weak. After initial training of the segmentation and aesthetic scoring models based on the aesthetic evaluation dataset, the segmentation model can be further optimized and trained based on the expanded sample set (i.e., the second sample images and their labeled segmentation labels) while keeping the parameters of other parts of the network fixed. This can lead to better image segmentation results and facilitate accurate bounding box generation. The training steps for the segmentation model based on the segmentation labels of the second sample images can refer to the training steps for the segmentation model based on the segmentation labels of the first sample images.

[0121] Among these optional implementations, when using the aesthetic evaluation dataset as the training set to train the segmentation model and the aesthetic scoring model, the training set of the segmentation model can be expanded after training to optimize the segmentation model separately and improve the segmentation accuracy of the segmentation model.

[0122] The technical solution of this disclosure includes: obtaining a first sample image, a segmentation label of the first sample image, and an aesthetic rating label corresponding to the sample cropping box of the first sample image; extracting features from the first sample image to obtain a third feature map; reconstructing the third feature map using a segmentation model to obtain a second segmented image; training the segmentation model based on the second segmented image and the segmentation label; generating a second candidate box within the first sample image; determining a fourth feature map corresponding to the second candidate box based on the third feature map and the second candidate box; outputting a predicted score for the fourth feature map using an aesthetic rating model; and training the aesthetic rating model based on the predicted score and the aesthetic rating label.

[0123] By training an image cropping model that includes a segmentation model and an aesthetic scoring model, the trained segmentation model can be used to determine the first segmented image in any image cropping method of this disclosure embodiment. Furthermore, by determining the bounding box of the target object in the image to be cropped based on the first segmented image, and generating first candidate boxes within the bounding boxes, the cropping interval can be narrowed, significantly reducing the number of candidate boxes generated. Finally, the trained aesthetic scoring model can be used to perform aesthetic scoring on the first feature map corresponding to each first candidate box, thereby achieving image cropping based on aesthetic scoring.

[0124] Example 5

[0125] Figure 8 This is a schematic diagram of an image cropping device according to Embodiment 5 of this disclosure. The image cropping device provided in this embodiment is applicable to image cropping, and is particularly suitable for cropping images with salient objects.

[0126] like Figure 8 As shown, the image cropping device provided in this embodiment may include:

[0127] The bounding box determination module 810 is used to segment the image to be cropped to obtain a first segmented image, and to determine the bounding box of the target object in the image to be cropped based on the first segmented image.

[0128] The target box determination module 820 is used to generate a first candidate box within the bounding box, and select a first target box from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box;

[0129] The cropping module 830 is used to take the image located within the first target box in the image to be cropped as the cropping result.

[0130] In some alternative implementations, the bounding box determination module may include:

[0131] The segmentation unit can be used to extract features from the image to be cropped to obtain a second feature map, and then reconstruct the features from the second feature map using a segmentation model to obtain a first segmented image.

[0132] Accordingly, the first feature map is determined based on the second feature map and the first candidate box.

[0133] In some alternative implementations, the bounding box determination module may include:

[0134] The bounding box determination unit can be used to determine the bounding box of the target object in the image to be cropped based on the position coordinates of the pixels belonging to the semantic classification of the target object in the first segmented image.

[0135] In some alternative implementations, the target bounding box determination module may include:

[0136] The candidate box generation unit can be used to generate a first candidate box that conforms to the input clipping ratio within the bounding box; and / or,

[0137] Based on the input clipping precision, generate a number of first candidate boxes within the bounding box corresponding to the clipping precision.

[0138] In some alternative implementations, the target bounding box determination module may also include:

[0139] The aesthetic scoring unit can be used to input the first feature map corresponding to the first candidate box into the aesthetic scoring model in batches according to the single processing volume of the aesthetic scoring model after the first candidate box is generated, so that the aesthetic scoring model outputs the aesthetic score of the first feature map.

[0140] In some alternative implementations, the target bounding box determination module may also include:

[0141] The preprocessing unit can be used to adjust the first feature map corresponding to the first candidate box to a preset size before inputting the first feature map corresponding to the first candidate box into the aesthetic scoring model in batches.

[0142] The image cropping method apparatus provided in this disclosure can execute the image cropping method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0143] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0144] Example 6

[0145] Figure 9 This is a schematic diagram of a model training device provided in Embodiment Six of this disclosure. The model training device provided in this embodiment is suitable for training an image cropping model that includes a segmentation model and an aesthetic scoring model.

[0146] like Figure 9 As shown, the model training apparatus provided in this embodiment may include:

[0147] The sample acquisition module 910 is used to acquire the first sample image, the segmentation label of the first sample image, and the aesthetic score label corresponding to the sample cropping box of the first sample image.

[0148] Feature extraction module 920 is used to extract features from the first sample image to obtain a third feature map;

[0149] The segmentation model training module 930 is used to reconstruct the second segmentation image from the third feature map using the segmentation model, and to train the segmentation model based on the second segmentation image and the segmentation label.

[0150] The candidate box feature determination module 940 is used to generate a second candidate box in the first sample image and determine the fourth feature map corresponding to the second candidate box based on the third feature map and the second candidate box.

[0151] The aesthetic scoring model training module 950 is used to output the predicted score of the fourth feature map through the aesthetic scoring model, and to train the aesthetic scoring model based on the predicted score and the aesthetic score label.

[0152] The trained segmentation model is used to determine the first segmented image in any image cropping method of this disclosure embodiment; the trained aesthetic scoring model is used to determine the aesthetic score in any image cropping method of this disclosure embodiment.

[0153] In some optional implementations, if the first sample image and the aesthetic rating label belong to the aesthetic evaluation dataset, the segmentation label is obtained by segmenting the first sample image based on a preset model;

[0154] Correspondingly, the segmentation model training module can also be used for:

[0155] Once the segmentation model and aesthetic scoring model have been trained, a second sample image is obtained, and segmentation labels are assigned to the second sample image.

[0156] With the parameters of the aesthetic scoring model fixed, the third segmentation image of the second sample image is determined by the trained segmentation model, and the segmentation model is optimized based on the segmentation labels of the third segmentation image and the second sample image.

[0157] The model training apparatus provided in this disclosure can execute the model training method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0158] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0159] Example 7

[0160] The following is for reference. Figure 10 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 10 The diagram below shows the structure of the terminal device or server 1000. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0161] like Figure 10 As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1006 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0162] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0163] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1006, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the image cropping method or model training method of embodiments of this disclosure.

[0164] The electronic device provided in this embodiment belongs to the same concept as the image cropping method or model training method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0165] Example 8

[0166] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the image cropping method or model training method provided in the above embodiments.

[0167] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution device or apparatus. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0168] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0169] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0170] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0171] The image to be cropped is segmented to obtain a first segmented image. The bounding box of the target object in the image to be cropped is determined based on the first segmented image. A first candidate box is generated within the bounding box. The first target box is selected from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box. The image in the image to be cropped that is located within the first target box is used as the cropping result.

[0172] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0173] The process involves obtaining a first sample image, its segmentation label, and an aesthetic rating label corresponding to the cropping bounding box of the first sample image; extracting features from the first sample image to obtain a third feature map; reconstructing the third feature map using a segmentation model to obtain a second segmented image; training the segmentation model based on the second segmented image and the segmentation label; generating a second candidate bounding box within the first sample image; determining a fourth feature map corresponding to the second candidate bounding box based on the third feature map and the second candidate bounding box; outputting a predicted score for the fourth feature map using an aesthetic rating model; and training the aesthetic rating model based on the predicted score and the aesthetic rating label. The trained segmentation model is used to determine the first segmented image in any image cropping method of this embodiment; the trained aesthetic rating model is used to determine the aesthetic score in any image cropping method of this embodiment.

[0174] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0176] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0177] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0178] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution apparatus or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0179] According to one or more embodiments of this disclosure, [Example 1] provides an image cropping method, the method comprising:

[0180] The image to be cropped is segmented to obtain a first segmented image, and the bounding box of the target object in the image to be cropped is determined based on the first segmented image.

[0181] A first candidate box is generated within the bounding box, and a first target box is selected from the first candidate box based on the aesthetic score of the first feature map corresponding to the first candidate box.

[0182] The image located within the first target box in the image to be cropped is taken as the cropping result.

[0183] According to one or more embodiments of this disclosure, [Example 2] provides an image cropping method, which further includes:

[0184] In some optional implementations, segmenting the image to be cropped to obtain a first segmented image includes:

[0185] The image to be cropped is subjected to feature extraction to obtain a second feature map, and the second feature map is reconstructed using a segmentation model to obtain a first segmented image;

[0186] Accordingly, the first feature map is determined based on the second feature map and the first candidate box.

[0187] According to one or more embodiments of this disclosure, [Example 3] provides an image cropping method, which further includes:

[0188] In some optional implementations, determining the bounding box of the target object in the image to be cropped based on the first segmented image includes:

[0189] Based on the position coordinates of the pixels belonging to the semantic classification of the target object in the first segmented image, the bounding box of the target object in the image to be cropped is determined.

[0190] According to one or more embodiments of this disclosure, [Example 4] provides an image cropping method, further comprising:

[0191] In some optional implementations, generating the first candidate box within the bounding box includes:

[0192] Based on the input clipping ratio, generate a first candidate box that conforms to the clipping ratio within the bounding box; and / or,

[0193] Based on the input clipping precision, a number of first candidate boxes corresponding to the clipping precision are generated within the bounding box.

[0194] According to one or more embodiments of this disclosure, [Example 5] provides an image cropping method and a model training method, further comprising:

[0195] In some alternative implementations, after generating the first candidate box, the following steps are also included:

[0196] Based on the single processing capacity of the aesthetic scoring model, the first feature map corresponding to the first candidate box is input into the aesthetic scoring model in batches so that the aesthetic scoring model outputs the aesthetic score of the first feature map.

[0197] According to one or more embodiments of this disclosure, [Example Six] provides an image cropping method and a model training method, further comprising:

[0198] In some optional implementations, before inputting the first feature map corresponding to the first candidate box into the aesthetic scoring model in batches, the method further includes:

[0199] Adjust the first feature map corresponding to the first candidate box to a preset size.

[0200] According to one or more embodiments of this disclosure, [Example Seven] provides a model training method, including:

[0201] Obtain the first sample image, the segmentation label of the first sample image, and the aesthetic rating label corresponding to the sample cropping box of the first sample image;

[0202] The first sample image is used to extract features to obtain a third feature map;

[0203] The third feature map is reconstructed using a segmentation model to obtain a second segmented image, and the segmentation model is trained based on the second segmented image and the segmentation label.

[0204] A second candidate box is generated within the first sample image, and a fourth feature map corresponding to the second candidate box is determined based on the third feature map and the second candidate box.

[0205] The aesthetic scoring model outputs the predicted score of the fourth feature map, and the aesthetic scoring model is trained based on the predicted score and the aesthetic score label.

[0206] The trained segmentation model is used to determine the first segmented image in any of the image cropping methods of claims 1-6; the trained aesthetic scoring model is used to determine the aesthetic score in any of the image cropping methods of claims 1-6.

[0207] According to one or more embodiments of this disclosure, [Example Eight] provides a model training method, further comprising:

[0208] In some optional implementations, if the first sample image and the aesthetic rating label belong to an aesthetic evaluation dataset, the segmentation label is obtained by segmenting the first sample image based on a preset model.

[0209] Accordingly, after the segmentation model and the aesthetic scoring model have been trained, the following steps are also included:

[0210] Obtain a second sample image and label the second sample image with segmentation labels;

[0211] With the parameters of the aesthetic scoring model fixed, the third segmentation image of the second sample image is determined by the trained segmentation model, and the segmentation model is optimized based on the third segmentation image and the segmentation labels of the second sample image.

[0212] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0213] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0214] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image cropping method, characterized in that, include: The image to be cropped is segmented to obtain a first segmented image. The bounding box of the target object in the image to be cropped is determined based on the first segmented image. The bounding box is a closed box that includes the entire target object and whose distance from the outline of the target object is greater than a first preset value and less than a second preset value. Multiple first candidate boxes are generated within the bounding box. Based on the aesthetic score of the first feature map corresponding to the first candidate box, the first candidate box with the highest aesthetic score is selected from the multiple first candidate boxes as the first target box. Alternatively, the multiple first candidate boxes are sorted according to the aesthetic score of each first feature map, and the first target box is determined according to the sorting result and the desired cropping range. The image located within the first target box in the image to be cropped is taken as the cropping result; Determining the bounding box of the target object in the image to be cropped based on the first segmented image includes: determining the bounding box of the target object in the image to be cropped based on the position coordinates of the pixels in the first segmented image that belong to the semantic classification of the target object.

2. The method according to claim 1, characterized in that, The process of segmenting the image to be cropped to obtain a first segmented image includes: The image to be cropped is subjected to feature extraction to obtain a second feature map, and the second feature map is reconstructed using a segmentation model to obtain a first segmented image; Accordingly, the first feature map is determined based on the second feature map and the first candidate box.

3. The method according to claim 1, characterized in that, The step of generating a first candidate box within the bounding box includes: Based on the input clipping ratio, generate a first candidate box that conforms to the clipping ratio within the bounding box; and / or, Based on the input clipping precision, a number of first candidate boxes corresponding to the clipping precision are generated within the bounding box.

4. The method according to claim 1, characterized in that, After generating the first candidate box, the method further includes: Based on the single processing capacity of the aesthetic scoring model, the first feature map corresponding to the first candidate box is input into the aesthetic scoring model in batches so that the aesthetic scoring model outputs the aesthetic score of the first feature map.

5. The method according to claim 4, characterized in that, Before inputting the first feature map corresponding to the first candidate box into the aesthetic scoring model in batches, the method further includes: Adjust the first feature map corresponding to the first candidate box to a preset size.

6. A model training method, characterized in that, include: Obtain the first sample image, the segmentation label of the first sample image, and the aesthetic rating label corresponding to the sample cropping box of the first sample image; The first sample image is used to extract features to obtain a third feature map; The third feature map is reconstructed using a segmentation model to obtain a second segmented image. A first loss value is calculated based on the second segmented image and the segmentation label. The segmentation model is then trained based on the first loss value, wherein the first loss value is calculated using a cross-entropy loss function. A second candidate box is generated within the first sample image, and a fourth feature map corresponding to the second candidate box is determined based on the third feature map and the second candidate box. The aesthetic scoring model outputs the predicted score of the fourth feature map, calculates the second loss value based on the predicted score and the aesthetic score label, and trains the aesthetic scoring model based on the second loss value, wherein the second loss value is calculated by a pixel-level smooth absolute value loss function. The trained segmentation model is used to determine the first segmented image in any of the image cropping methods of claims 1-5; the trained aesthetic scoring model is used to determine the aesthetic score in any of the image cropping methods of claims 1-5, and the segmentation model and the aesthetic scoring model can be jointly trained based on the sum of the first loss value and the second loss value.

7. The model training method according to claim 6, characterized in that, If the first sample image and the aesthetic rating label belong to the aesthetic evaluation dataset, then the segmentation label is obtained by segmenting the first sample image based on a preset model; Accordingly, after the segmentation model and the aesthetic scoring model have been trained, the following steps are also included: Obtain a second sample image and label the second sample image with segmentation labels; With the parameters of the aesthetic scoring model fixed, the third segmentation image of the second sample image is determined by the trained segmentation model, and the segmentation model is optimized based on the third segmentation image and the segmentation labels of the second sample image.

8. An image cropping device, characterized in that, include: The bounding box determination module is used to segment the image to be cropped to obtain a first segmented image, and to determine the bounding box of the target object in the image to be cropped based on the first segmented image. The bounding box is a closed box that includes the entire target object and whose distance from the outline of the target object is greater than a first preset value and less than a second preset value. The target box determination module is used to generate multiple first candidate boxes within the bounding box, select the first candidate box with the highest aesthetic score from the multiple first candidate boxes as the first target box based on the aesthetic score of the first feature map corresponding to the first candidate box, or sort the multiple first candidate boxes according to the aesthetic score of each first feature map, and determine the first target box based on the sorting result and the desired clipping range. The cropping module is used to take the image located within the first target box in the image to be cropped as the cropping result; The bounding box determination module includes: a border determination unit, used to determine the bounding box of the target object in the image to be cropped based on the position coordinates of the pixels belonging to the semantic classification of the target object in the first segmented image.

9. A model training device, characterized in that, include: The sample acquisition module is used to acquire a first sample image, the segmentation label of the first sample image, and the aesthetic rating label corresponding to the sample cropping box of the first sample image. The feature extraction module is used to extract features from the first sample image to obtain a third feature map; The segmentation model training module is used to reconstruct the third feature map using a segmentation model to obtain a second segmentation image, calculate a first loss value based on the second segmentation image and the segmentation label, and train the segmentation model based on the first loss value, wherein the first loss value is calculated using the cross-entropy loss function; The candidate box feature determination module is used to generate a second candidate box in the first sample image and determine a fourth feature map corresponding to the second candidate box based on the third feature map and the second candidate box. An aesthetic scoring model training module is used to output the predicted score of the fourth feature map through the aesthetic scoring model, calculate a second loss value based on the predicted score and the aesthetic scoring label, and train the aesthetic scoring model based on the second loss value, wherein the second loss value is calculated by a pixel-level smooth absolute value loss function; The trained segmentation model is used to determine the first segmented image in any of the image cropping methods of claims 1-5; the trained aesthetic scoring model is used to determine the aesthetic score in any of the image cropping methods of claims 1-5, and the segmentation model and the aesthetic scoring model can be jointly trained based on the sum of the first loss value and the second loss value.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image cropping method as described in any one of claims 1-5, or the model training method as described in any one of claims 6-7.

11. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image cropping method as described in any one of claims 1-5, or to implement the model training method as described in any one of claims 6-7.

Images